From cda5910ab08614e4b9e25b612148e47b87dd1247 Mon Sep 17 00:00:00 2001 From: SHL0MS Date: Fri, 13 Mar 2026 19:13:12 -0400 Subject: [PATCH] update ascii-video skill: design patterns, local time, examples - New references/design-patterns.md: layer hierarchy (bg/content/accent), directional parameter arcs, scene concepts and visual metaphors, counter-rotating systems, wave collision, progressive fragmentation, entropy/consumption, staggered crescendo buildup, scene ordering - New references/examples.md: copy-paste-ready scenes at every complexity - Update scenes.md: local time convention (t=0 at scene start) - Update SKILL.md: add design-patterns.md to reference table - Add README.md to hermes-agent copy - Sync all reference docs with canonical source (SHL0MS/ascii-video) --- skills/creative/ascii-video/README.md | 249 ++++ skills/creative/ascii-video/SKILL.md | 26 +- .../ascii-video/references/architecture.md | 314 ++++- .../ascii-video/references/composition.md | 288 ++++- .../ascii-video/references/design-patterns.md | 193 +++ .../ascii-video/references/effects.md | 1110 ++++++++++++++++- .../ascii-video/references/examples.md | 416 ++++++ .../creative/ascii-video/references/inputs.md | 371 +++++- .../ascii-video/references/optimization.md | 265 +++- .../creative/ascii-video/references/scenes.md | 26 +- .../ascii-video/references/shaders.md | 330 +++++ .../ascii-video/references/troubleshooting.md | 10 + 12 files changed, 3503 insertions(+), 95 deletions(-) create mode 100644 skills/creative/ascii-video/README.md create mode 100644 skills/creative/ascii-video/references/design-patterns.md create mode 100644 skills/creative/ascii-video/references/examples.md diff --git a/skills/creative/ascii-video/README.md b/skills/creative/ascii-video/README.md new file mode 100644 index 000000000..678930cfa --- /dev/null +++ b/skills/creative/ascii-video/README.md @@ -0,0 +1,249 @@ +# ☤ ASCII Video + +Renders any content as colored ASCII character video. Audio, video, images, text, or pure math in, MP4/GIF/PNG sequence out. Full RGB color per character cell, 1080p 24fps default. No GPU. + +Built for [Hermes Agent](https://github.com/NousResearch/hermes-agent). Usable in any coding agent. Canonical source lives here; synced to [`NousResearch/hermes-agent/skills/creative/ascii-video`](https://github.com/NousResearch/hermes-agent/tree/main/skills/creative/ascii-video) via PR. + +## What this is + +A skill that teaches an agent how to build single-file Python renderers for ASCII video from scratch. The agent gets the full pipeline: grid system, font rasterization, effect library, shader chain, audio analysis, parallel encoding. It writes the renderer, runs it, gets video. + +The output is actual video. Not terminal escape codes. Frames are computed as grids of colored characters, composited onto pixel canvases with pre-rasterized font bitmaps, post-processed through shaders, piped to ffmpeg. + +## Modes + +| Mode | Input | Output | +|------|-------|--------| +| Video-to-ASCII | A video file | ASCII recreation of the footage | +| Audio-reactive | An audio file | Visuals driven by frequency bands, beats, energy | +| Generative | Nothing | Procedural animation from math | +| Hybrid | Video + audio | ASCII video with audio-reactive overlays | +| Lyrics/text | Audio + timed text (SRT) | Karaoke-style text with effects | +| TTS narration | Text quotes + API key | Narrated video with typewriter text and generated speech | + +## Pipeline + +Every mode follows the same 6-stage path: + +``` +INPUT --> ANALYZE --> SCENE_FN --> TONEMAP --> SHADE --> ENCODE +``` + +1. **Input** loads source material (or nothing for generative). +2. **Analyze** extracts per-frame features. Audio gets 6-band FFT, RMS, spectral centroid, flatness, flux, beat detection with exponential decay. Video gets luminance, edges, motion. +3. **Scene function** returns a pixel canvas directly. Composes multiple character grids at different densities, value/hue fields, pixel blend modes. This is where the visuals happen. +4. **Tonemap** does adaptive percentile-based brightness normalization with per-scene gamma. ASCII on black is inherently dark. Linear multipliers don't work. This does. +5. **Shade** runs a `ShaderChain` (38 composable shaders) plus a `FeedbackBuffer` for temporal recursion with spatial transforms. +6. **Encode** pipes raw RGB frames to ffmpeg for H.264 encoding. Segments concatenated, audio muxed. + +## Grid system + +Characters render on fixed-size grids. Layer multiple densities for depth. + +| Size | Font | Grid at 1080p | Use | +|------|------|---------------|-----| +| xs | 8px | 400x108 | Ultra-dense data fields | +| sm | 10px | 320x83 | Rain, starfields | +| md | 16px | 192x56 | Default balanced | +| lg | 20px | 160x45 | Readable text | +| xl | 24px | 137x37 | Large titles | +| xxl | 40px | 80x22 | Giant minimal | + +Rendering the same scene on `sm` and `lg` then screen-blending them creates natural texture interference. Fine detail shows through gaps in coarse characters. Most scenes use two or three grids. + +## Character palettes (20+) + +Each sorted dark-to-bright, each a different visual texture. Validated against the font at init so broken glyphs get dropped silently. + +| Family | Examples | Feel | +|--------|----------|------| +| Density ramps | ` .:-=+#@█` | Classic ASCII art gradient | +| Block elements | ` ░▒▓█▄▀▐▌` | Chunky, digital | +| Braille | ` ⠁⠂⠃...⠿` | Fine-grained pointillism | +| Dots | ` ⋅∘∙●◉◎` | Smooth, organic | +| Stars | ` ·✧✦✩✨★✶` | Sparkle, celestial | +| Half-fills | ` ◔◑◕◐◒◓◖◗◙` | Directional fill progression | +| Crosshatch | ` ▣▤▥▦▧▨▩` | Hatched density ramp | +| Math | ` ·∘∙•°±×÷≈≠≡∞∫∑Ω` | Scientific, abstract | +| Box drawing | ` ─│┌┐└┘├┤┬┴┼` | Structural, circuit-like | +| Katakana | ` ·ｦｧｨｩｪｫｬｭ...` | Matrix rain | +| Greek | ` αβγδεζηθ...ω` | Classical, academic | +| Runes | ` ᚠᚢᚦᚱᚷᛁᛇᛒᛖᛚᛞᛟ` | Mystical, ancient | +| Alchemical | ` ☉☽♀♂♃♄♅♆♇` | Esoteric | +| Arrows | ` ←↑→↓↔↕↖↗↘↙` | Directional, kinetic | +| Music | ` ♪♫♬♩♭♮♯○●` | Musical | +| Project-specific | ` .·~=≈∞⚡☿✦★⊕◊◆▲▼●■` | Themed per project | + +Custom palettes are built per project to match the content. + +## Color strategies + +| Strategy | How it maps hue | Good for | +|----------|----------------|----------| +| Angle-mapped | Position angle from center | Rainbow radial effects | +| Distance-mapped | Distance from center | Depth, tunnels | +| Frequency-mapped | Audio spectral centroid | Timbral shifting | +| Value-mapped | Brightness level | Heat maps, fire | +| Time-cycled | Slow rotation over time | Ambient, chill | +| Source-sampled | Original video pixel colors | Video-to-ASCII | +| Palette-indexed | Discrete lookup table | Retro, flat graphic | +| Temperature | Warm-to-cool blend | Emotional tone | +| Complementary | Hue + opposite | Bold, dramatic | +| Triadic | Three equidistant hues | Psychedelic, vibrant | +| Analogous | Neighboring hues | Harmonious, subtle | +| Monochrome | Fixed hue, vary S/V | Noir, focused | + +Plus 10 discrete RGB palettes (neon, pastel, cyberpunk, vaporwave, earth, ice, blood, forest, mono-green, mono-amber). + +## Effects + +### Backgrounds + +| Effect | Description | Parameters | +|--------|-------------|------------| +| Sine field | Layered sinusoidal interference | freq, speed, octave count | +| Smooth noise | Multi-octave Perlin approximation | octaves, scale | +| Cellular | Voronoi-like moving cells | n_centers, speed | +| Noise/static | Random per-cell flicker | density | +| Video source | Downsampled video frame | brightness | + +### Primary effects + +| Effect | Description | +|--------|-------------| +| Concentric rings | Bass-driven pulsing rings with wobble | +| Radial rays | Spoke pattern, beat-triggered | +| Spiral arms | Logarithmic spiral, configurable arm count/tightness | +| Tunnel | Infinite depth perspective | +| Vortex | Twisting radial distortion | +| Frequency waves | Per-band sine waves at different heights | +| Interference | Overlapping sine waves creating moire | +| Aurora | Horizontal flowing bands | +| Ripple | Point-source concentric waves | +| Fire columns | Rising flames with heat-color gradient | +| Spectrum bars | Mirrored frequency visualizer | +| Waveform | Oscilloscope-style trace | + +### Particle systems + +| Type | Behavior | Character sets | +|------|----------|---------------| +| Explosion | Beat-triggered radial burst | `*+#@⚡✦★█▓` | +| Sparks | Short-lived bright dots | `·•●★✶*+` | +| Embers | Rising from bottom with drift | `·•●★` | +| Snow | Falling with wind sway | `❄❅❆·•*○` | +| Rain | Fast vertical streaks | `│┃║/\` | +| Bubbles | Rising, expanding | `○◎◉●∘∙°` | +| Data | Falling hex/binary | `01{}[]<>/\` | +| Runes | Mystical floating symbols | `ᚠᚢᚦᚱᚷᛁ✦★` | +| Orbit | Circular/elliptical paths | `·•●` | +| Gravity well | Attracted to point sources | configurable | +| Dissolve | Spread across screen, fade | configurable | +| Starfield | 3D projected, approaching | configurable | + +## Shader pipeline + +38 composable shaders, applied to the pixel canvas after character rendering. Configurable per section. + +| Category | Shaders | +|----------|---------| +| Geometry | CRT barrel, pixelate, wave distort, displacement map, kaleidoscope, mirror (h/v/quad/diag) | +| Channel | Chromatic aberration (beat-reactive), channel shift, channel swap, RGB split radial | +| Color | Invert, posterize, threshold, solarize, hue rotate, saturation, color grade, color wobble, color ramp | +| Glow/Blur | Bloom, edge glow, soft focus, radial blur | +| Noise | Film grain (beat-reactive), static noise | +| Lines/Patterns | Scanlines, halftone | +| Tone | Vignette, contrast, gamma, levels, brightness | +| Glitch/Data | Glitch bands (beat-reactive), block glitch, pixel sort, data bend | + +12 color tint presets: warm, cool, matrix green, amber, sepia, neon pink, ice, blood, forest, void, sunset, neutral. + +7 mood presets for common shader combos: + +| Mood | Shaders | +|------|---------| +| Retro terminal | CRT + scanlines + grain + amber/green tint | +| Clean modern | Light bloom + subtle vignette | +| Glitch art | Heavy chromatic + glitch bands + color wobble | +| Cinematic | Bloom + vignette + grain + color grade | +| Dreamy | Heavy bloom + soft focus + color wobble | +| Harsh/industrial | High contrast + grain + scanlines, no bloom | +| Psychedelic | Color wobble + chromatic + kaleidoscope mirror | + +## Blend modes and composition + +20 pixel blend modes for layering canvases: normal, add, subtract, multiply, screen, overlay, softlight, hardlight, difference, exclusion, colordodge, colorburn, linearlight, vividlight, pin_light, hard_mix, lighten, darken, grain_extract, grain_merge. + +Mirror modes: horizontal, vertical, quad, diagonal, kaleidoscope (6-fold radial). Beat-triggered. + +Transitions: crossfade, directional wipe, radial wipe, dissolve, glitch cut. + +## Hardware adaptation + +Auto-detects CPU count, RAM, platform, ffmpeg. Adapts worker count, resolution, FPS. + +| Profile | Resolution | FPS | When | +|---------|-----------|-----|------| +| `draft` | 960x540 | 12 | Check timing/layout | +| `preview` | 1280x720 | 15 | Review effects | +| `production` | 1920x1080 | 24 | Final output | +| `max` | 3840x2160 | 30 | Ultra-high | +| `auto` | Detected | 24 | Adapts to hardware + duration | + +`auto` estimates render time and downgrades if it would take over an hour. Low-memory systems drop to 720p automatically. + +### Render times (1080p 24fps, ~180ms/frame/worker) + +| Duration | 4 workers | 8 workers | 16 workers | +|----------|-----------|-----------|------------| +| 30s | ~3 min | ~2 min | ~1 min | +| 2 min | ~13 min | ~7 min | ~4 min | +| 5 min | ~33 min | ~17 min | ~9 min | +| 10 min | ~65 min | ~33 min | ~17 min | + +720p roughly halves these. 4K roughly quadruples them. + +## Known pitfalls + +**Brightness.** ASCII characters are small bright dots on black. Most frame pixels are background. Linear `* N` multipliers clip highlights and wash out. Use `tonemap()` with per-scene gamma instead. Default gamma 0.75, solarize scenes 0.55, posterize 0.50. + +**Render bottleneck.** The per-cell Python loop compositing font bitmaps runs at ~100-150ms/frame. Unavoidable without Cython/C. Everything else must be vectorized numpy. Python for-loops over rows/cols in effect functions will tank performance. + +**ffmpeg deadlock.** Never `stderr=subprocess.PIPE` on long-running encodes. Buffer fills at ~64KB, process hangs. Redirect stderr to a file. + +**Font cell height.** Pillow's `textbbox()` returns wrong height on macOS. Use `font.getmetrics()` for `ascent + descent`. + +**Font compatibility.** Not all Unicode renders in all fonts. Palettes validated at init, blank glyphs silently removed. + +## Requirements + +◆ Python 3.10+ +◆ NumPy, Pillow, SciPy (audio modes) +◆ ffmpeg on PATH +◆ A monospace font (Menlo, Courier, Monaco, auto-detected) +◆ Optional: OpenCV, ElevenLabs API key (TTS mode) + +## File structure + +``` +├── SKILL.md # Modes, workflow, creative direction +├── README.md # This file +└── references/ + ├── architecture.md # Grid system, fonts, palettes, color, _render_vf() + ├── effects.md # Value fields, hue fields, backgrounds, particles + ├── shaders.md # 38 shaders, ShaderChain, tint presets, transitions + ├── composition.md # Blend modes, multi-grid, tonemap, FeedbackBuffer + ├── scenes.md # Scene protocol, SCENES table, render_clip(), examples + ├── design-patterns.md # Layer hierarchy, directional arcs, scene concepts + ├── inputs.md # Audio analysis, video sampling, text, TTS + ├── optimization.md # Hardware detection, vectorized patterns, parallelism + └── troubleshooting.md # Broadcasting traps, blend pitfalls, diagnostics +``` + +## Projects built with this + +✦ 85-second highlight reel. 15 scenes (14×5s + 15s crescendo finale), randomized order, directional parameter arcs, layer hierarchy composition. Showcases the full effect vocabulary: fBM, voronoi fragmentation, reaction-diffusion, cellular automata, dual counter-rotating spirals, wave collision, domain warping, tunnel descent, kaleidoscope symmetry, boid flocking, fire simulation, glitch corruption, and a 7-layer crescendo buildup. + +✦ Audio-reactive music visualizer. 3.5 min, 8 sections with distinct effects, beat-triggered particles and glitch, cycling palettes. + +✦ TTS narrated testimonial video. 23 quotes, per-quote ElevenLabs voices, background music at 15% wide stereo, per-clip re-rendering for iterative editing. diff --git a/skills/creative/ascii-video/SKILL.md b/skills/creative/ascii-video/SKILL.md index 8c686bf23..83e2d0f66 100644 --- a/skills/creative/ascii-video/SKILL.md +++ b/skills/creative/ascii-video/SKILL.md @@ -59,16 +59,20 @@ Every mode follows the same 6-stage pipeline. See `references/architecture.md` f | Dimension | Options | Reference | |-----------|---------|-----------| | **Character palette** | Density ramps, block elements, symbols, scripts (katakana, Greek, runes, braille), dots, project-specific | `architecture.md` § Character Palettes | -| **Color strategy** | HSV (angle/distance/time/value mapped), discrete RGB palettes, monochrome, complementary, triadic, temperature | `architecture.md` § Color System | +| **Color strategy** | HSV (angle/distance/time/value mapped), OKLAB/OKLCH (perceptually uniform), discrete RGB palettes, auto-generated harmony (complementary/triadic/analogous/tetradic), monochrome, temperature | `architecture.md` § Color System | | **Color tint** | Warm, cool, amber, matrix green, neon pink, sepia, ice, blood, void, sunset | `shaders.md` § Color Grade | -| **Background texture** | Sine fields, noise, smooth noise, cellular/voronoi, video source | `effects.md` § Background Fills | -| **Primary effects** | Rings, spirals, tunnel, vortex, waves, interference, aurora, ripple, fire | `effects.md` § Radial / Wave / Fire | -| **Particles** | Energy sparks, snow, rain, bubbles, runes, binary data, orbits, gravity wells | `effects.md` § Particle Systems | +| **Background texture** | Sine fields, fBM noise, domain warp, voronoi cells, reaction-diffusion, cellular automata, video source | `effects.md` § Background Fills, Noise-Based Fields, Simulation-Based Fields | +| **Primary effects** | Rings, spirals, tunnel, vortex, waves, interference, aurora, ripple, fire, strange attractors, SDFs (geometric shapes with smooth booleans) | `effects.md` § Radial / Wave / Fire / SDF-Based Fields | +| **Particles** | Energy sparks, snow, rain, bubbles, runes, binary data, orbits, gravity wells, flocking boids, flow-field followers, trail-drawing particles | `effects.md` § Particle Systems | | **Shader mood** | Retro CRT, clean modern, glitch art, cinematic, dreamy, harsh industrial, psychedelic | `shaders.md` § Design Philosophy | | **Grid density** | xs(8px) through xxl(40px), mixed per layer | `architecture.md` § Grid System | | **Font** | Menlo, Monaco, Courier, SF Mono, JetBrains Mono, Fira Code, IBM Plex | `architecture.md` § Font Selection | +| **Coordinate space** | Cartesian, polar, tiled, rotated, skewed, fisheye, twisted, Möbius, domain-warped | `effects.md` § Coordinate Transforms | | **Mirror mode** | None, horizontal, vertical, quad, diagonal, kaleidoscope | `shaders.md` § Mirror Effects | -| **Transition style** | Crossfade, wipe (directional/radial), dissolve, glitch cut | `shaders.md` § Transitions | +| **Masking** | Circle, rect, ring, gradient, text stencil, value-field-as-mask, animated iris/wipe/dissolve | `composition.md` § Masking | +| **Temporal motion** | Static, audio-reactive, eased keyframes, morphing between fields, temporal noise (smooth in-place evolution) | `effects.md` § Temporal Coherence | +| **Transition style** | Crossfade, wipe (directional/radial), dissolve, glitch cut, iris open/close, mask-based reveal | `shaders.md` § Transitions, `composition.md` § Animated Masks | +| **Aspect ratio** | Landscape (16:9), portrait (9:16), square (1:1), ultrawide (21:9) | `architecture.md` § Resolution Presets | ### Per-Section Variation @@ -95,10 +99,11 @@ Establish with user: - **Input source** — file path, format, duration - **Mode** — which of the 6 modes above - **Sections** — time-mapped style changes (timestamps → effect names) -- **Resolution** — default 1920x1080 @ 24fps; GIFs typically 640x360 @ 15fps +- **Resolution** — landscape 1920x1080 (default), portrait 1080x1920, square 1080x1080 @ 24fps; GIFs typically 640x360 @ 15fps - **Style direction** — dense/sparse, bright/dark, chaotic/minimal, color palette - **Text/branding** — easter eggs, overlays, credits, themed character sets - **Output format** — MP4 (default), GIF, PNG sequence +- **Aspect ratio** — landscape (16:9), portrait (9:16 for TikTok/Reels/Stories), square (1:1 for IG feed) ### Step 2: Detect Hardware and Set Quality @@ -240,11 +245,12 @@ Image.fromarray(canvas).save("test.png") | File | Contents | |------|----------| -| `references/architecture.md` | Grid system, font selection, character palettes (library of 20+), color system (HSV + discrete RGB), `_render_vf()` helper, compositing, v2 effect function contract | +| `references/architecture.md` | Grid system (landscape/portrait/square resolution presets), font selection, character palettes (library of 20+), color system (HSV + OKLAB/OKLCH + discrete RGB + color harmony generation + perceptual gradient interpolation), `_render_vf()` helper, compositing, v2 effect function contract | | `references/inputs.md` | All input sources: audio analysis, video sampling, image conversion, text/lyrics, TTS integration (ElevenLabs, voice assignment, audio mixing) | -| `references/effects.md` | Effect building blocks: 12 value field generators (`vf_sinefield` through `vf_noise_static`), 8 hue field generators (`hf_fixed` through `hf_plasma`), radial/wave/fire effects, particles, composing guide | +| `references/effects.md` | Effect building blocks: 20+ value field generators (trig, noise/fBM, domain warp, voronoi, reaction-diffusion, cellular automata, strange attractors, SDFs), 8 hue field generators, coordinate transforms (rotate/tile/polar/Möbius), temporal coherence (easing, keyframes, morphing), radial/wave/fire effects, advanced particles (flocking, flow fields, trails), composing guide | | `references/shaders.md` | 38 shader implementations (geometry, channel, color, glow, noise, pattern, tone, glitch, mirror), `ShaderChain` class, full `_apply_shader_step()` dispatch, audio-reactive scaling, transitions, tint presets | -| `references/composition.md` | **v2 core**: pixel blend modes (20 modes with implementations), multi-grid composition, `_render_vf()` helper, adaptive `tonemap()`, per-scene gamma, `FeedbackBuffer` with spatial transforms, `PixelBlendStack` | -| `references/scenes.md` | **v2 scene protocol**: scene function contract, `Renderer` class, `SCENES` table structure, `render_clip()` loop, beat-synced cutting, parallel rendering + pickling constraints, 4 complete scene examples, scene design checklist | +| `references/composition.md` | **v2 core**: pixel blend modes (20 modes with implementations), multi-grid composition, `_render_vf()` helper, adaptive `tonemap()`, per-scene gamma, `FeedbackBuffer` with spatial transforms, `PixelBlendStack`, masking/stencil system (shape masks, text stencils, animated masks, boolean ops) | +| `references/scenes.md` | **v2 scene protocol**: scene function contract (local time convention), `Renderer` class, `SCENES` table structure, `render_clip()` loop, beat-synced cutting, parallel rendering + pickling constraints, 4 complete scene examples, scene design checklist | +| `references/design-patterns.md` | **Scene composition patterns**: layer hierarchy (bg/content/accent), directional parameter arcs vs oscillation, scene concepts and visual metaphors, counter-rotating dual systems, wave collision, progressive fragmentation, entropy/consumption, staggered layer entry (crescendo), scene ordering | | `references/troubleshooting.md` | NumPy broadcasting traps, blend mode pitfalls, multiprocessing/pickling issues, brightness diagnostics, ffmpeg deadlocks, font issues, performance bottlenecks, common mistakes | | `references/optimization.md` | Hardware detection, adaptive quality profiles (draft/preview/production/max), CLI integration, vectorized effect patterns, parallel rendering, memory management | diff --git a/skills/creative/ascii-video/references/architecture.md b/skills/creative/ascii-video/references/architecture.md index a255523a3..5a6c3c9d4 100644 --- a/skills/creative/ascii-video/references/architecture.md +++ b/skills/creative/ascii-video/references/architecture.md @@ -1,12 +1,43 @@ # Architecture Reference +**Cross-references:** +- Effect building blocks (value fields, noise, SDFs, particles): `effects.md` +- `_render_vf()`, blend modes, tonemap, masking: `composition.md` +- Scene protocol, render_clip, SCENES table: `scenes.md` +- Shader pipeline, feedback buffer, output encoding: `shaders.md` +- Complete scene examples: `examples.md` +- Input sources (audio analysis, video, TTS): `inputs.md` +- Performance tuning, hardware detection: `optimization.md` +- Common bugs (broadcasting, font, encoding): `troubleshooting.md` + ## Grid System +### Resolution Presets + +```python +RESOLUTION_PRESETS = { + "landscape": (1920, 1080), # 16:9 — YouTube, default + "portrait": (1080, 1920), # 9:16 — TikTok, Reels, Stories + "square": (1080, 1080), # 1:1 — Instagram feed + "ultrawide": (2560, 1080), # 21:9 — cinematic + "landscape4k":(3840, 2160), # 16:9 — 4K + "portrait4k": (2160, 3840), # 9:16 — 4K portrait +} + +def get_resolution(preset="landscape", custom=None): + """Returns (VW, VH) tuple.""" + if custom: + return custom + return RESOLUTION_PRESETS.get(preset, RESOLUTION_PRESETS["landscape"]) +``` + ### Multi-Density Grids -Pre-initialize multiple grid sizes. Switch per section for visual variety. +Pre-initialize multiple grid sizes. Switch per section for visual variety. Grid dimensions auto-compute from resolution: -| Key | Font Size | Grid (1920x1080) | Use | +**Landscape (1920x1080):** + +| Key | Font Size | Grid (cols x rows) | Use | |-----|-----------|-------------------|-----| | xs | 8 | 400x108 | Ultra-dense data fields | | sm | 10 | 320x83 | Dense detail, rain, starfields | @@ -15,7 +46,34 @@ Pre-initialize multiple grid sizes. Switch per section for visual variety. | xl | 24 | 137x37 | Short quotes, large titles | | xxl | 40 | 80x22 | Giant text, minimal | -**Grid sizing for text-heavy content**: When displaying readable text (quotes, lyrics, testimonials), use 20px (`lg`) as the primary grid. This gives 160 columns -- plenty for lines up to ~50 chars centered. For very short quotes (< 60 chars, <= 3 lines), 24px (`xl`) makes them more impactful. Only init the grids you actually use -- each grid pre-rasterizes all characters which costs ~0.3-0.5s. +**Portrait (1080x1920):** + +| Key | Font Size | Grid (cols x rows) | Use | +|-----|-----------|-------------------|-----| +| xs | 8 | 225x192 | Ultra-dense, tall data columns | +| sm | 10 | 180x148 | Dense detail, vertical rain | +| md | 16 | 112x100 | Default balanced | +| lg | 20 | 90x80 | Readable text (~30 chars/line centered) | +| xl | 24 | 75x66 | Short quotes, stacked | +| xxl | 40 | 45x39 | Giant text, minimal | + +**Square (1080x1080):** + +| Key | Font Size | Grid (cols x rows) | Use | +|-----|-----------|-------------------|-----| +| sm | 10 | 180x83 | Dense detail | +| md | 16 | 112x56 | Default balanced | +| lg | 20 | 90x45 | Readable text | + +**Key differences in portrait mode:** +- Fewer columns (90 at `lg` vs 160) — lines must be shorter or wrap +- Many more rows (80 at `lg` vs 45) — vertical stacking is natural +- Aspect ratio correction flips: `asp = cw / ch` still works but the visual emphasis is vertical +- Radial effects appear as tall ellipses unless corrected +- Vertical effects (rain, embers, fire columns) are naturally enhanced +- Horizontal effects (spectrum bars, waveforms) need rotation or compression + +**Grid sizing for text in portrait**: Use `lg` (20px) for 2-3 word lines. Max comfortable line length is ~25-30 chars. For longer quotes, break aggressively into many short lines stacked vertically — portrait has vertical space to spare. `xl` (24px) works for single words or very short phrases. Grid dimensions: `cols = VW // cell_width`, `rows = VH // cell_height`. @@ -59,7 +117,23 @@ FONT_PREFS_LINUX = [ ("Noto Sans Mono", "/usr/share/fonts/truetype/noto/NotoSansMono-Regular.ttf"), ("Ubuntu Mono", "/usr/share/fonts/truetype/ubuntu/UbuntuMono-R.ttf"), ] -FONT_PREFS = FONT_PREFS_MACOS if platform.system() == "Darwin" else FONT_PREFS_LINUX +FONT_PREFS_WINDOWS = [ + ("Consolas", r"C:\Windows\Fonts\consola.ttf"), + ("Courier New", r"C:\Windows\Fonts\cour.ttf"), + ("Lucida Console", r"C:\Windows\Fonts\lucon.ttf"), + ("Cascadia Code", os.path.expandvars(r"%LOCALAPPDATA%\Microsoft\Windows\Fonts\CascadiaCode.ttf")), + ("Cascadia Mono", os.path.expandvars(r"%LOCALAPPDATA%\Microsoft\Windows\Fonts\CascadiaMono.ttf")), +] + +def _get_font_prefs(): + s = platform.system() + if s == "Darwin": + return FONT_PREFS_MACOS + elif s == "Windows": + return FONT_PREFS_WINDOWS + return FONT_PREFS_LINUX + +FONT_PREFS = _get_font_prefs() ``` **Multi-font rendering**: use different fonts for different layers (e.g., monospace for background, a bolder variant for overlay text). Each GridLayer owns its own font: @@ -77,8 +151,8 @@ Before initializing grids, gather all characters that need bitmap pre-rasterizat all_chars = set() for pal in [PAL_DEFAULT, PAL_DENSE, PAL_BLOCKS, PAL_RUNE, PAL_KATA, PAL_GREEK, PAL_MATH, PAL_DOTS, PAL_BRAILLE, PAL_STARS, - PAL_BINARY, PAL_MUSIC, PAL_BOX, PAL_CIRCUIT, PAL_ARROWS, - PAL_HERMES]: # ... all palettes used in project + PAL_HALFFILL, PAL_HATCH, PAL_BINARY, PAL_MUSIC, PAL_BOX, + PAL_CIRCUIT, PAL_ARROWS, PAL_HERMES]: # ... all palettes used in project all_chars.update(pal) # Add any overlay text characters all_chars.update("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 .,-:;!?/|") @@ -87,21 +161,31 @@ all_chars.discard(" ") # space is never rendered ### GridLayer Initialization -Each grid pre-computes coordinate arrays for vectorized effect math: +Each grid pre-computes coordinate arrays for vectorized effect math. The grid automatically adapts to any resolution (landscape, portrait, square): ```python class GridLayer: - def __init__(self, font_path, font_size): + def __init__(self, font_path, font_size, vw=None, vh=None): + """Initialize grid for any resolution. + vw, vh: video width/height in pixels. Defaults to global VW, VH.""" + vw = vw or VW; vh = vh or VH + self.vw = vw; self.vh = vh + self.font = ImageFont.truetype(font_path, font_size) asc, desc = self.font.getmetrics() bbox = self.font.getbbox("M") self.cw = bbox[2] - bbox[0] # character cell width self.ch = asc + desc # CRITICAL: not textbbox height - self.cols = VW // self.cw - self.rows = VH // self.ch - self.ox = (VW - self.cols * self.cw) // 2 # centering - self.oy = (VH - self.rows * self.ch) // 2 + self.cols = vw // self.cw + self.rows = vh // self.ch + self.ox = (vw - self.cols * self.cw) // 2 # centering + self.oy = (vh - self.rows * self.ch) // 2 + + # Aspect ratio metadata + self.aspect = vw / vh # >1 = landscape, <1 = portrait, 1 = square + self.is_portrait = vw < vh + self.is_landscape = vw > vh # Index arrays self.rr = np.arange(self.rows, dtype=np.float32)[:, None] @@ -219,9 +303,11 @@ PAL_ARABIC = " \u0627\u0628\u062a\u062b\u062c\u062d\u062e\u062f\u0630\u0631\u0 #### Dot / Point Progressions ```python -PAL_DOTS = " \u22c5\u2218\u2219\u25cf\u25c9\u25ce\u25c6\u2726\u2605" # dot size progression -PAL_BRAILLE = " \u2801\u2802\u2803\u2804\u2805\u2806\u2807\u2808\u2809\u280a\u280b\u280c\u280d\u280e\u280f\u2810\u2811\u2812\u2813\u2814\u2815\u2816\u2817\u2818\u2819\u281a\u281b\u281c\u281d\u281e\u281f\u283f" # braille patterns -PAL_STARS = " \u00b7\u2727\u2726\u2729\u2728\u2605\u2736\u2733\u2738" # star progression +PAL_DOTS = " ⋅∘∙●◉◎◆✦★" # dot size progression +PAL_BRAILLE = " ⠁⠂⠃⠄⠅⠆⠇⠈⠉⠊⠋⠌⠍⠎⠏⠐⠑⠒⠓⠔⠕⠖⠗⠘⠙⠚⠛⠜⠝⠞⠟⠿" # braille patterns +PAL_STARS = " ·✧✦✩✨★✶✳✸" # star progression +PAL_HALFFILL = " ◔◑◕◐◒◓◖◗◙" # directional half-fill progression +PAL_HATCH = " ▣▤▥▦▧▨▩" # crosshatch density ramp ``` #### Project-Specific (examples -- invent new ones per project) @@ -353,6 +439,202 @@ def rgb_palette_map(val, mask, palette): return R, G, B ``` +### OKLAB Color Space (Perceptually Uniform) + +HSV hue is perceptually non-uniform: green occupies far more visual range than blue. OKLAB / OKLCH provide perceptually even color steps — hue increments of 0.1 look equally different regardless of starting hue. Use OKLAB for: +- Gradient interpolation (no unwanted intermediate hues) +- Color harmony generation (perceptually balanced palettes) +- Smooth color transitions over time + +```python +# --- sRGB <-> Linear sRGB --- + +def srgb_to_linear(c): + """Convert sRGB [0,1] to linear light. c: float32 array.""" + return np.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4) + +def linear_to_srgb(c): + """Convert linear light to sRGB [0,1].""" + return np.where(c <= 0.0031308, c * 12.92, 1.055 * np.power(np.maximum(c, 0), 1/2.4) - 0.055) + +# --- Linear sRGB <-> OKLAB --- + +def linear_rgb_to_oklab(r, g, b): + """Linear sRGB to OKLAB. r,g,b: float32 arrays [0,1]. + Returns (L, a, b) where L=[0,1], a,b=[-0.4, 0.4] approx.""" + l_ = 0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b + m_ = 0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b + s_ = 0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b + l_c = np.cbrt(l_); m_c = np.cbrt(m_); s_c = np.cbrt(s_) + L = 0.2104542553 * l_c + 0.7936177850 * m_c - 0.0040720468 * s_c + a = 1.9779984951 * l_c - 2.4285922050 * m_c + 0.4505937099 * s_c + b_ = 0.0259040371 * l_c + 0.7827717662 * m_c - 0.8086757660 * s_c + return L, a, b_ + +def oklab_to_linear_rgb(L, a, b): + """OKLAB to linear sRGB. Returns (r, g, b) float32 arrays [0,1].""" + l_ = L + 0.3963377774 * a + 0.2158037573 * b + m_ = L - 0.1055613458 * a - 0.0638541728 * b + s_ = L - 0.0894841775 * a - 1.2914855480 * b + l_c = l_ ** 3; m_c = m_ ** 3; s_c = s_ ** 3 + r = +4.0767416621 * l_c - 3.3077115913 * m_c + 0.2309699292 * s_c + g = -1.2684380046 * l_c + 2.6097574011 * m_c - 0.3413193965 * s_c + b_ = -0.0041960863 * l_c - 0.7034186147 * m_c + 1.7076147010 * s_c + return np.clip(r, 0, 1), np.clip(g, 0, 1), np.clip(b_, 0, 1) + +# --- Convenience: sRGB uint8 <-> OKLAB --- + +def rgb_to_oklab(R, G, B): + """sRGB uint8 arrays to OKLAB.""" + r = srgb_to_linear(R.astype(np.float32) / 255.0) + g = srgb_to_linear(G.astype(np.float32) / 255.0) + b = srgb_to_linear(B.astype(np.float32) / 255.0) + return linear_rgb_to_oklab(r, g, b) + +def oklab_to_rgb(L, a, b): + """OKLAB to sRGB uint8 arrays.""" + r, g, b_ = oklab_to_linear_rgb(L, a, b) + R = np.clip(linear_to_srgb(r) * 255, 0, 255).astype(np.uint8) + G = np.clip(linear_to_srgb(g) * 255, 0, 255).astype(np.uint8) + B = np.clip(linear_to_srgb(b_) * 255, 0, 255).astype(np.uint8) + return R, G, B + +# --- OKLCH (cylindrical form of OKLAB) --- + +def oklab_to_oklch(L, a, b): + """OKLAB to OKLCH. Returns (L, C, H) where H is in [0, 1] (normalized).""" + C = np.sqrt(a**2 + b**2) + H = (np.arctan2(b, a) / (2 * np.pi)) % 1.0 + return L, C, H + +def oklch_to_oklab(L, C, H): + """OKLCH to OKLAB. H in [0, 1].""" + angle = H * 2 * np.pi + a = C * np.cos(angle) + b = C * np.sin(angle) + return L, a, b +``` + +### Gradient Interpolation (OKLAB vs HSV) + +Interpolating colors through OKLAB avoids the hue detours that HSV produces: + +```python +def lerp_oklab(color_a, color_b, t_array): + """Interpolate between two sRGB colors through OKLAB. + color_a, color_b: (R, G, B) tuples 0-255 + t_array: float32 array [0,1] — interpolation parameter per pixel. + Returns (R, G, B) uint8 arrays.""" + La, aa, ba = rgb_to_oklab( + np.full_like(t_array, color_a[0], dtype=np.uint8), + np.full_like(t_array, color_a[1], dtype=np.uint8), + np.full_like(t_array, color_a[2], dtype=np.uint8)) + Lb, ab, bb = rgb_to_oklab( + np.full_like(t_array, color_b[0], dtype=np.uint8), + np.full_like(t_array, color_b[1], dtype=np.uint8), + np.full_like(t_array, color_b[2], dtype=np.uint8)) + L = La + (Lb - La) * t_array + a = aa + (ab - aa) * t_array + b = ba + (bb - ba) * t_array + return oklab_to_rgb(L, a, b) + +def lerp_oklch(color_a, color_b, t_array, short_path=True): + """Interpolate through OKLCH (preserves chroma, smooth hue path). + short_path: take the shorter arc around the hue wheel.""" + La, aa, ba = rgb_to_oklab( + np.full_like(t_array, color_a[0], dtype=np.uint8), + np.full_like(t_array, color_a[1], dtype=np.uint8), + np.full_like(t_array, color_a[2], dtype=np.uint8)) + Lb, ab, bb = rgb_to_oklab( + np.full_like(t_array, color_b[0], dtype=np.uint8), + np.full_like(t_array, color_b[1], dtype=np.uint8), + np.full_like(t_array, color_b[2], dtype=np.uint8)) + L1, C1, H1 = oklab_to_oklch(La, aa, ba) + L2, C2, H2 = oklab_to_oklch(Lb, ab, bb) + # Shortest hue path + if short_path: + dh = H2 - H1 + dh = np.where(dh > 0.5, dh - 1.0, np.where(dh < -0.5, dh + 1.0, dh)) + H = (H1 + dh * t_array) % 1.0 + else: + H = H1 + (H2 - H1) * t_array + L = L1 + (L2 - L1) * t_array + C = C1 + (C2 - C1) * t_array + Lout, aout, bout = oklch_to_oklab(L, C, H) + return oklab_to_rgb(Lout, aout, bout) +``` + +### Color Harmony Generation + +Auto-generate harmonious palettes from a seed color: + +```python +def harmony_complementary(seed_rgb): + """Two colors: seed + opposite hue.""" + L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]])) + _, C, H = oklab_to_oklch(L, a, b) + return [seed_rgb, _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.5) % 1.0)] + +def harmony_triadic(seed_rgb): + """Three colors: seed + two at 120-degree offsets.""" + L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]])) + _, C, H = oklab_to_oklch(L, a, b) + return [seed_rgb, + _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.333) % 1.0), + _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.667) % 1.0)] + +def harmony_analogous(seed_rgb, spread=0.08, n=5): + """N colors spread evenly around seed hue.""" + L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]])) + _, C, H = oklab_to_oklch(L, a, b) + offsets = np.linspace(-spread * (n-1)/2, spread * (n-1)/2, n) + return [_oklch_to_srgb_tuple(L[0], C[0], (H[0] + off) % 1.0) for off in offsets] + +def harmony_split_complementary(seed_rgb, split=0.08): + """Three colors: seed + two flanking the complement.""" + L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]])) + _, C, H = oklab_to_oklch(L, a, b) + comp = (H[0] + 0.5) % 1.0 + return [seed_rgb, + _oklch_to_srgb_tuple(L[0], C[0], (comp - split) % 1.0), + _oklch_to_srgb_tuple(L[0], C[0], (comp + split) % 1.0)] + +def harmony_tetradic(seed_rgb): + """Four colors: two complementary pairs at 90-degree offset.""" + L, a, b = rgb_to_oklab(np.array([seed_rgb[0]]), np.array([seed_rgb[1]]), np.array([seed_rgb[2]])) + _, C, H = oklab_to_oklch(L, a, b) + return [seed_rgb, + _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.25) % 1.0), + _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.5) % 1.0), + _oklch_to_srgb_tuple(L[0], C[0], (H[0] + 0.75) % 1.0)] + +def _oklch_to_srgb_tuple(L, C, H): + """Helper: single OKLCH -> sRGB (R,G,B) int tuple.""" + La = np.array([L]); Ca = np.array([C]); Ha = np.array([H]) + Lo, ao, bo = oklch_to_oklab(La, Ca, Ha) + R, G, B = oklab_to_rgb(Lo, ao, bo) + return (int(R[0]), int(G[0]), int(B[0])) +``` + +### OKLAB Hue Fields + +Drop-in replacements for `hf_*` generators that produce perceptually uniform hue variation: + +```python +def hf_oklch_angle(offset=0.0, chroma=0.12, lightness=0.7): + """OKLCH hue mapped to angle from center. Perceptually uniform rainbow. + Returns (R, G, B) uint8 color array instead of a float hue. + NOTE: Use with _render_vf_rgb() variant, not standard _render_vf().""" + def fn(g, f, t, S): + H = (g.angle / (2 * np.pi) + offset + t * 0.05) % 1.0 + L = np.full_like(H, lightness) + C = np.full_like(H, chroma) + Lo, ao, bo = oklch_to_oklab(L, C, H) + R, G, B = oklab_to_rgb(Lo, ao, bo) + return mkc(R, G, B, g.rows, g.cols) + return fn +``` + ### Compositing Helpers ```python @@ -458,7 +740,7 @@ subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", concat_path, ### v2 Protocol (Current) -Every scene function: `(renderer, features_dict, time_float, state_dict) -> canvas_uint8` +Every scene function: `(r, f, t, S) -> canvas_uint8` — where `r` = Renderer, `f` = features dict, `t` = time float, `S` = persistent state dict ```python def fx_example(r, f, t, S): diff --git a/skills/creative/ascii-video/references/composition.md b/skills/creative/ascii-video/references/composition.md index 17e3088f2..d23ffe3e2 100644 --- a/skills/creative/ascii-video/references/composition.md +++ b/skills/creative/ascii-video/references/composition.md @@ -1,6 +1,14 @@ # Composition & Brightness Reference -The composable system is the core of visual complexity. It operates at three levels: pixel-level blend modes, multi-grid composition, and adaptive brightness management. This document covers all three. +The composable system is the core of visual complexity. It operates at three levels: pixel-level blend modes, multi-grid composition, and adaptive brightness management. This document covers all three, plus the masking/stencil system for spatial control. + +**Cross-references:** +- Grid system, palettes, color (HSV + OKLAB): `architecture.md` +- Effect building blocks (value fields, hue fields, particles): `effects.md` +- Scene protocol, render_clip, SCENES table: `scenes.md` +- Shader pipeline, feedback buffer: `shaders.md` +- Complete scene examples with blend/mask usage: `examples.md` +- Blend mode pitfalls (overlay crush, division by zero): `troubleshooting.md` ## Pixel-Level Blend Modes @@ -102,6 +110,69 @@ result = blend_canvas(result, canvas_c, "difference", 0.6) Order matters: `screen(A, B)` is commutative, but `difference(screen(A,B), C)` differs from `difference(A, screen(B,C))`. +### Linear-Light Blend Modes + +Standard `blend_canvas()` operates in sRGB space — the raw byte values. This is fine for most uses, but sRGB is perceptually non-linear: blending in sRGB darkens midtones and shifts hues slightly. For physically accurate blending (matching how light actually combines), convert to linear light first. + +Uses `srgb_to_linear()` / `linear_to_srgb()` from `architecture.md` § OKLAB Color System. + +```python +def blend_canvas_linear(base, top, mode="normal", opacity=1.0): + """Blend in linear light space for physically accurate results. + + Identical API to blend_canvas(), but converts sRGB → linear before + blending and linear → sRGB after. More expensive (~2x) due to the + gamma conversions, but produces correct results for additive blending, + screen, and any mode where brightness matters. + """ + af = srgb_to_linear(base.astype(np.float32) / 255.0) + bf = srgb_to_linear(top.astype(np.float32) / 255.0) + fn = BLEND_MODES.get(mode, BLEND_MODES["normal"]) + result = fn(af, bf) + if opacity < 1.0: + result = af * (1 - opacity) + result * opacity + result = linear_to_srgb(np.clip(result, 0, 1)) + return np.clip(result * 255, 0, 255).astype(np.uint8) +``` + +**When to use `blend_canvas_linear()` vs `blend_canvas()`:** + +| Scenario | Use | Why | +|----------|-----|-----| +| Screen-blending two bright layers | `linear` | sRGB screen over-brightens highlights | +| Add mode for glow/bloom effects | `linear` | Additive light follows linear physics | +| Blending text overlay at low opacity | `srgb` | Perceptual blending looks more natural for text | +| Multiply for shadow/darkening | `srgb` | Differences are minimal for darken ops | +| Color-critical work (matching reference) | `linear` | Avoids sRGB hue shifts in midtones | +| Performance-critical inner loop | `srgb` | ~2x faster, good enough for most ASCII art | + +**Batch version** for compositing many layers (converts once, blends multiple, converts back): + +```python +def blend_many_linear(layers, modes, opacities): + """Blend a stack of layers in linear light space. + + Args: + layers: list of uint8 (H,W,3) canvases + modes: list of blend mode strings (len = len(layers) - 1) + opacities: list of floats (len = len(layers) - 1) + Returns: + uint8 (H,W,3) canvas + """ + # Convert all to linear at once + linear = [srgb_to_linear(l.astype(np.float32) / 255.0) for l in layers] + result = linear[0] + for i in range(1, len(linear)): + fn = BLEND_MODES.get(modes[i-1], BLEND_MODES["normal"]) + blended = fn(result, linear[i]) + op = opacities[i-1] + if op < 1.0: + blended = result * (1 - op) + blended * op + result = np.clip(blended, 0, 1) + result = linear_to_srgb(result) + return np.clip(result * 255, 0, 255).astype(np.uint8) +``` + --- ## Multi-Grid Composition @@ -219,19 +290,22 @@ def tonemap(canvas, target_mean=90, gamma=0.75, black_point=2, white_point=253): """Adaptive tone-mapping: normalizes + gamma-corrects so no frame is fully dark or washed out. - 1. Compute 1st and 99.5th percentile (ignores outlier pixels) + 1. Compute 1st and 99.5th percentile on 4x subsample (16x fewer values, + negligible accuracy loss, major speedup at 1080p+) 2. Stretch that range to [0, 1] 3. Apply gamma curve (< 1 lifts shadows, > 1 darkens) 4. Rescale to [black_point, white_point] """ f = canvas.astype(np.float32) - lo = np.percentile(f, 1) - hi = np.percentile(f, 99.5) + sub = f[::4, ::4] # 4x subsample: ~390K values vs ~6.2M at 1080p + lo = np.percentile(sub, 1) + hi = np.percentile(sub, 99.5) if hi - lo < 10: hi = max(hi, lo + 10) # near-uniform frame fallback f = np.clip((f - lo) / (hi - lo), 0.0, 1.0) - f = np.power(f, gamma) - f = f * (white_point - black_point) + black_point + np.power(f, gamma, out=f) # in-place: avoids allocation + np.multiply(f, (white_point - black_point), out=f) + np.add(f, black_point, out=f) return np.clip(f, 0, 255).astype(np.uint8) ``` @@ -453,6 +527,208 @@ class FeedbackBuffer: --- +## Masking / Stencil System + +Masks are float32 arrays `(rows, cols)` or `(VH, VW)` in range [0, 1]. They control where effects are visible: 1.0 = fully visible, 0.0 = fully hidden. Use masks to create figure/ground relationships, focal points, and shaped reveals. + +### Shape Masks + +```python +def mask_circle(g, cx_frac=0.5, cy_frac=0.5, radius=0.3, feather=0.05): + """Circular mask centered at (cx_frac, cy_frac) in normalized coords. + feather: width of soft edge (0 = hard cutoff).""" + asp = g.cw / g.ch if hasattr(g, 'cw') else 1.0 + dx = (g.cc / g.cols - cx_frac) + dy = (g.rr / g.rows - cy_frac) * asp + d = np.sqrt(dx**2 + dy**2) + if feather > 0: + return np.clip(1.0 - (d - radius) / feather, 0, 1) + return (d <= radius).astype(np.float32) + +def mask_rect(g, x0=0.2, y0=0.2, x1=0.8, y1=0.8, feather=0.03): + """Rectangular mask. Coordinates in [0,1] normalized.""" + dx = np.maximum(x0 - g.cc / g.cols, g.cc / g.cols - x1) + dy = np.maximum(y0 - g.rr / g.rows, g.rr / g.rows - y1) + d = np.maximum(dx, dy) + if feather > 0: + return np.clip(1.0 - d / feather, 0, 1) + return (d <= 0).astype(np.float32) + +def mask_ring(g, cx_frac=0.5, cy_frac=0.5, inner_r=0.15, outer_r=0.35, + feather=0.03): + """Ring / annulus mask.""" + inner = mask_circle(g, cx_frac, cy_frac, inner_r, feather) + outer = mask_circle(g, cx_frac, cy_frac, outer_r, feather) + return outer - inner + +def mask_gradient_h(g, start=0.0, end=1.0): + """Left-to-right gradient mask.""" + return np.clip((g.cc / g.cols - start) / (end - start + 1e-10), 0, 1).astype(np.float32) + +def mask_gradient_v(g, start=0.0, end=1.0): + """Top-to-bottom gradient mask.""" + return np.clip((g.rr / g.rows - start) / (end - start + 1e-10), 0, 1).astype(np.float32) + +def mask_gradient_radial(g, cx_frac=0.5, cy_frac=0.5, inner=0.0, outer=0.5): + """Radial gradient mask — bright at center, dark at edges.""" + d = np.sqrt((g.cc / g.cols - cx_frac)**2 + (g.rr / g.rows - cy_frac)**2) + return np.clip(1.0 - (d - inner) / (outer - inner + 1e-10), 0, 1) +``` + +### Value Field as Mask + +Use any `vf_*` function's output as a spatial mask: + +```python +def mask_from_vf(vf_result, threshold=0.5, feather=0.1): + """Convert a value field to a mask by thresholding. + feather: smooth edge width around threshold.""" + if feather > 0: + return np.clip((vf_result - threshold + feather) / (2 * feather), 0, 1) + return (vf_result > threshold).astype(np.float32) + +def mask_select(mask, vf_a, vf_b): + """Spatial conditional: show vf_a where mask is 1, vf_b where mask is 0. + mask: float32 [0,1] array. Intermediate values blend.""" + return vf_a * mask + vf_b * (1 - mask) +``` + +### Text Stencil + +Render text to a mask. Effects are visible only through the letterforms: + +```python +def mask_text(grid, text, row_frac=0.5, font=None, font_size=None): + """Render text string as a float32 mask [0,1] at grid resolution. + Characters = 1.0, background = 0.0. + + row_frac: vertical position as fraction of grid height. + font: PIL ImageFont (defaults to grid's font if None). + font_size: override font size for the mask text (for larger stencil text). + """ + from PIL import Image, ImageDraw, ImageFont + + f = font or grid.font + if font_size and font != grid.font: + f = ImageFont.truetype(font.path, font_size) + + # Render text to image at pixel resolution, then downsample to grid + img = Image.new("L", (grid.cols * grid.cw, grid.ch), 0) + draw = ImageDraw.Draw(img) + bbox = draw.textbbox((0, 0), text, font=f) + tw = bbox[2] - bbox[0] + x = (grid.cols * grid.cw - tw) // 2 + draw.text((x, 0), text, fill=255, font=f) + row_mask = np.array(img, dtype=np.float32) / 255.0 + + # Place in full grid mask + mask = np.zeros((grid.rows, grid.cols), dtype=np.float32) + target_row = int(grid.rows * row_frac) + # Downsample rendered text to grid cells + for c in range(grid.cols): + px = c * grid.cw + if px + grid.cw <= row_mask.shape[1]: + cell = row_mask[:, px:px + grid.cw] + if cell.mean() > 0.1: + mask[target_row, c] = cell.mean() + return mask + +def mask_text_block(grid, lines, start_row_frac=0.3, font=None): + """Multi-line text stencil. Returns full grid mask.""" + mask = np.zeros((grid.rows, grid.cols), dtype=np.float32) + for i, line in enumerate(lines): + row_frac = start_row_frac + i / grid.rows + line_mask = mask_text(grid, line, row_frac, font) + mask = np.maximum(mask, line_mask) + return mask +``` + +### Animated Masks + +Masks that change over time for reveals, wipes, and morphing: + +```python +def mask_iris(g, t, t_start, t_end, cx_frac=0.5, cy_frac=0.5, + max_radius=0.7, ease_fn=None): + """Iris open/close: circle that grows from 0 to max_radius. + ease_fn: easing function (default: ease_in_out_cubic from effects.md).""" + if ease_fn is None: + ease_fn = lambda x: x * x * (3 - 2 * x) # smoothstep fallback + progress = np.clip((t - t_start) / (t_end - t_start), 0, 1) + radius = ease_fn(progress) * max_radius + return mask_circle(g, cx_frac, cy_frac, radius, feather=0.03) + +def mask_wipe_h(g, t, t_start, t_end, direction="right"): + """Horizontal wipe reveal.""" + progress = np.clip((t - t_start) / (t_end - t_start), 0, 1) + if direction == "left": + progress = 1 - progress + return mask_gradient_h(g, start=progress - 0.05, end=progress + 0.05) + +def mask_wipe_v(g, t, t_start, t_end, direction="down"): + """Vertical wipe reveal.""" + progress = np.clip((t - t_start) / (t_end - t_start), 0, 1) + if direction == "up": + progress = 1 - progress + return mask_gradient_v(g, start=progress - 0.05, end=progress + 0.05) + +def mask_dissolve(g, t, t_start, t_end, seed=42): + """Random pixel dissolve — noise threshold sweeps from 0 to 1.""" + progress = np.clip((t - t_start) / (t_end - t_start), 0, 1) + rng = np.random.RandomState(seed) + noise = rng.random((g.rows, g.cols)).astype(np.float32) + return (noise < progress).astype(np.float32) +``` + +### Mask Boolean Operations + +```python +def mask_union(a, b): + """OR — visible where either mask is active.""" + return np.maximum(a, b) + +def mask_intersect(a, b): + """AND — visible only where both masks are active.""" + return np.minimum(a, b) + +def mask_subtract(a, b): + """A minus B — visible where A is active but B is not.""" + return np.clip(a - b, 0, 1) + +def mask_invert(m): + """NOT — flip mask.""" + return 1.0 - m +``` + +### Applying Masks to Canvases + +```python +def apply_mask_canvas(canvas, mask, bg_canvas=None): + """Apply a grid-resolution mask to a pixel canvas. + Expands mask from (rows, cols) to (VH, VW) via nearest-neighbor. + + canvas: uint8 (VH, VW, 3) + mask: float32 (rows, cols) [0,1] + bg_canvas: what shows through where mask=0. None = black. + """ + # Expand mask to pixel resolution + mask_px = np.repeat(np.repeat(mask, canvas.shape[0] // mask.shape[0] + 1, axis=0), + canvas.shape[1] // mask.shape[1] + 1, axis=1) + mask_px = mask_px[:canvas.shape[0], :canvas.shape[1]] + + if bg_canvas is not None: + return np.clip(canvas * mask_px[:, :, None] + + bg_canvas * (1 - mask_px[:, :, None]), 0, 255).astype(np.uint8) + return np.clip(canvas * mask_px[:, :, None], 0, 255).astype(np.uint8) + +def apply_mask_vf(vf_a, vf_b, mask): + """Apply mask at value-field level — blend two value fields spatially. + All arrays are (rows, cols) float32.""" + return vf_a * mask + vf_b * (1 - mask) +``` + +--- + ## PixelBlendStack Higher-level wrapper for multi-layer compositing: diff --git a/skills/creative/ascii-video/references/design-patterns.md b/skills/creative/ascii-video/references/design-patterns.md new file mode 100644 index 000000000..2eff15e99 --- /dev/null +++ b/skills/creative/ascii-video/references/design-patterns.md @@ -0,0 +1,193 @@ +# Scene Design Patterns + +**Cross-references:** +- Scene protocol, SCENES table: `scenes.md` +- Blend modes, multi-grid composition, tonemap: `composition.md` +- Effect building blocks (value fields, noise, SDFs): `effects.md` +- Shader pipeline, feedback buffer: `shaders.md` +- Complete scene examples: `examples.md` + +Higher-order patterns for composing scenes that feel intentional rather than random. These patterns use the existing building blocks (value fields, blend modes, shaders, feedback) but organize them with compositional intent. + +## Layer Hierarchy + +Every scene should have clear visual layers with distinct roles: + +| Layer | Grid | Brightness | Purpose | +|-------|------|-----------|---------| +| **Background** | xs or sm (dense) | 0.1–0.25 | Atmosphere, texture. Never competes with content. | +| **Content** | md (balanced) | 0.4–0.8 | The main visual idea. Carries the scene's concept. | +| **Accent** | lg or sm (sparse) | 0.5–1.0 (sparse coverage) | Highlights, punctuation, sparse bright points. | + +The background sets mood. The content layer is what the scene *is about*. The accent adds visual interest without overwhelming. + +```python +def fx_example(r, f, t, S): + local = t + progress = min(local / 5.0, 1.0) + + g_bg = r.get_grid("sm") + g_main = r.get_grid("md") + g_accent = r.get_grid("lg") + + # --- Background: dim atmosphere --- + bg_val = vf_smooth_noise(g_bg, f, t * 0.3, S, octaves=2, bri=0.15) + # ... render bg to canvas + + # --- Content: the main visual idea --- + content_val = vf_spiral(g_main, f, t, S, n_arms=n_arms, tightness=tightness) + # ... render content on top of canvas + + # --- Accent: sparse highlights --- + accent_val = vf_noise_static(g_accent, f, t, S, density=0.05) + # ... render accent on top + + return canvas +``` + +## Directional Parameter Arcs + +Parameters should *go somewhere* over the scene's duration — not oscillate aimlessly with `sin(t * N)`. + +**Bad:** `twist = 3.0 + 2.0 * math.sin(t * 0.6)` — wobbles back and forth, feels aimless. + +**Good:** `twist = 2.0 + progress * 5.0` — starts gentle, ends intense. The scene *builds*. + +Use `progress = min(local / duration, 1.0)` (0→1 over the scene) to drive directional change: + +| Pattern | Formula | Feel | +|---------|---------|------| +| Linear ramp | `progress * range` | Steady buildup | +| Ease-out | `1 - (1 - progress) ** 2` | Fast start, gentle finish | +| Ease-in | `progress ** 2` | Slow start, accelerating | +| Step reveal | `np.clip((progress - 0.5) / 0.25, 0, 1)` | Nothing until 50%, then fades in | +| Build + plateau | `min(1.0, progress * 1.5)` | Reaches full at 67%, holds | + +Oscillation is fine for *secondary* parameters (saturation shimmer, hue drift). But the *defining* parameter of the scene should have a direction. + +### Examples of Directional Arcs + +| Scene concept | Parameter | Arc | +|--------------|-----------|-----| +| Emergence | Ring radius | 0 → max (ease-out) | +| Shatter | Voronoi cell count | 8 → 38 (linear) | +| Descent | Tunnel speed | 2.0 → 10.0 (linear) | +| Mandala | Shape complexity | ring → +polygon → +star → +rosette (step reveals) | +| Crescendo | Layer count | 1 → 7 (staggered entry) | +| Entropy | Geometry visibility | 1.0 → 0.0 (consumed) | + +## Scene Concepts + +Each scene should be built around a *visual idea*, not an effect name. + +**Bad:** "fx_plasma_cascade" — named after the effect. No concept. +**Good:** "fx_emergence" — a point of light expands into a field. The name tells you *what happens*. + +Good scene concepts have: +1. A **visual metaphor** (emergence, descent, collision, entropy) +2. A **directional arc** (things change from A to B, not oscillate) +3. **Motivated layer choices** (each layer serves the concept) +4. **Motivated feedback** (transform direction matches the metaphor) + +| Concept | Metaphor | Feedback transform | Why | +|---------|----------|-------------------|-----| +| Emergence | Birth, expansion | zoom-out | Past frames expand outward | +| Descent | Falling, acceleration | zoom-in | Past frames rush toward center | +| Inferno | Rising fire | shift-up | Past frames rise with the flames | +| Entropy | Decay, dissolution | none | Clean, no persistence — things disappear | +| Crescendo | Accumulation | zoom + hue_shift | Everything compounds and shifts | + +## Compositional Techniques + +### Counter-Rotating Dual Systems + +Two instances of the same effect rotating in opposite directions create visual interference: + +```python +# Primary spiral (clockwise) +s1_val = vf_spiral(g_main, f, t * 1.5, S, n_arms=n_arms_1, tightness=tightness_1) + +# Counter-rotating spiral (counter-clockwise via negative time) +s2_val = vf_spiral(g_accent, f, -t * 1.2, S, n_arms=n_arms_2, tightness=tightness_2) + +# Screen blend creates bright interference at crossing points +canvas = blend_canvas(canvas_with_s1, c2, "screen", 0.7) +``` + +Works with spirals, vortexes, rings. The counter-rotation creates constantly shifting interference patterns. + +### Wave Collision + +Two wave fronts converging from opposite sides, meeting at a collision point: + +```python +collision_phase = abs(progress - 0.5) * 2 # 1→0→1 (0 at collision) + +# Wave A approaches from left +offset_a = (1 - progress) * g.cols * 0.4 +wave_a = np.sin((g.cc + offset_a) * 0.08 + t * 2) * 0.5 + 0.5 + +# Wave B approaches from right +offset_b = -(1 - progress) * g.cols * 0.4 +wave_b = np.sin((g.cc + offset_b) * 0.08 - t * 2) * 0.5 + 0.5 + +# Interference peaks at collision +combined = wave_a * 0.5 + wave_b * 0.5 + np.abs(wave_a - wave_b) * (1 - collision_phase) * 0.5 +``` + +### Progressive Fragmentation + +Voronoi with cell count increasing over time — visual shattering: + +```python +n_pts = int(8 + progress * 30) # 8 cells → 38 cells +# Pre-generate enough points, slice to n_pts +px = base_x[:n_pts] + np.sin(t * 0.3 + np.arange(n_pts) * 0.7) * (3 + progress * 3) +``` + +The edge glow width can also increase with progress to emphasize the cracks. + +### Entropy / Consumption + +A clean geometric pattern being overtaken by an organic process: + +```python +# Geometry fades out +geo_val = clean_pattern * max(0.05, 1.0 - progress * 0.9) + +# Organic process grows in +rd_val = vf_reaction_diffusion(g, f, t, S) * min(1.0, progress * 1.5) + +# Render geometry first, organic on top — organic consumes geometry +``` + +### Staggered Layer Entry (Crescendo) + +Layers enter one at a time, building to overwhelming density: + +```python +def layer_strength(enter_t, ramp=1.5): + """0.0 until enter_t, ramps to 1.0 over ramp seconds.""" + return max(0.0, min(1.0, (local - enter_t) / ramp)) + +# Layer 1: always present +s1 = layer_strength(0.0) +# Layer 2: enters at 2s +s2 = layer_strength(2.0) +# Layer 3: enters at 4s +s3 = layer_strength(4.0) +# ... etc + +# Each layer uses a different effect, grid, palette, and blend mode +# Screen blend between layers so they accumulate light +``` + +For a 15-second crescendo, 7 layers entering every 2 seconds works well. Use different blend modes (screen for most, add for energy, colordodge for the final wash). + +## Scene Ordering + +For a multi-scene reel or video: +- **Vary mood between adjacent scenes** — don't put two calm scenes next to each other +- **Randomize order** rather than grouping by type — prevents "effect demo" feel +- **End on the strongest scene** — crescendo or something with a clear payoff +- **Open with energy** — grab attention in the first 2 seconds diff --git a/skills/creative/ascii-video/references/effects.md b/skills/creative/ascii-video/references/effects.md index ee0ff2c26..2242b95b4 100644 --- a/skills/creative/ascii-video/references/effects.md +++ b/skills/creative/ascii-video/references/effects.md @@ -1,6 +1,14 @@ # Effect Catalog -Effect building blocks that produce visual patterns. In v2, these are used **inside scene functions** that return a pixel canvas directly. The building blocks below operate on grid coordinate arrays and produce `(chars, colors)` or value/hue fields that the scene function renders to canvas via `_render_vf()`. See `composition.md` for the v2 rendering pattern and `scenes.md` for scene function examples. +Effect building blocks that produce visual patterns. In v2, these are used **inside scene functions** that return a pixel canvas directly. The building blocks below operate on grid coordinate arrays and produce `(chars, colors)` or value/hue fields that the scene function renders to canvas via `_render_vf()`. + +**Cross-references:** +- Grid system, palettes, color: `architecture.md` +- `_render_vf()`, blend modes, tonemap, masking: `composition.md` +- Scene protocol, render_clip, SCENES table: `scenes.md` +- Shader pipeline, feedback buffer: `shaders.md` +- Complete scene examples using these effects: `examples.md` +- Common bugs (broadcasting, clipping): `troubleshooting.md` ## Design Philosophy @@ -243,9 +251,9 @@ def eff_ripple(g, f, t, sources=None, freq=0.3, damping=0.02): ## Particle Systems ### General Pattern -All particle systems use persistent state: +All particle systems use persistent state via the `S` dict parameter: ```python -S = state # dict persisted across frames +# S is the persistent state dict (same as r.S, passed explicitly) if "px" not in S: S["px"]=[]; S["py"]=[]; S["vx"]=[]; S["vy"]=[]; S["life"]=[]; S["char"]=[] @@ -341,41 +349,268 @@ def emit_orbit(S, n=20, radius=15, speed=1.0, char_set=PART_DOT): # Particles that reach well center respawn at edges ``` +### Flocking / Boids + +Emergent swarm behavior from three simple rules: separation, alignment, cohesion. + +```python +def update_boids(S, g, f, n_boids=200, perception=8.0, max_speed=2.0, + sep_weight=1.5, ali_weight=1.0, coh_weight=1.0, + char_set=None): + """Boids flocking simulation. Particles self-organize into organic groups. + + perception: how far each boid can see (grid cells) + sep_weight: separation (avoid crowding) strength + ali_weight: alignment (match neighbor velocity) strength + coh_weight: cohesion (steer toward group center) strength + """ + if char_set is None: + char_set = list("·•●◦∘⬤") + if "boid_x" not in S: + rng = np.random.RandomState(42) + S["boid_x"] = rng.uniform(0, g.cols, n_boids).astype(np.float32) + S["boid_y"] = rng.uniform(0, g.rows, n_boids).astype(np.float32) + S["boid_vx"] = (rng.random(n_boids).astype(np.float32) - 0.5) * max_speed + S["boid_vy"] = (rng.random(n_boids).astype(np.float32) - 0.5) * max_speed + S["boid_ch"] = [random.choice(char_set) for _ in range(n_boids)] + + bx = S["boid_x"]; by = S["boid_y"] + bvx = S["boid_vx"]; bvy = S["boid_vy"] + n = len(bx) + + # For each boid, compute steering forces + ax = np.zeros(n, dtype=np.float32) + ay = np.zeros(n, dtype=np.float32) + + # Spatial hash for efficient neighbor lookup + cell_size = perception + cells = {} + for i in range(n): + cx_i = int(bx[i] / cell_size) + cy_i = int(by[i] / cell_size) + key = (cx_i, cy_i) + if key not in cells: + cells[key] = [] + cells[key].append(i) + + for i in range(n): + cx_i = int(bx[i] / cell_size) + cy_i = int(by[i] / cell_size) + sep_x, sep_y = 0.0, 0.0 + ali_x, ali_y = 0.0, 0.0 + coh_x, coh_y = 0.0, 0.0 + count = 0 + + # Check neighboring cells + for dcx in range(-1, 2): + for dcy in range(-1, 2): + for j in cells.get((cx_i + dcx, cy_i + dcy), []): + if j == i: + continue + dx = bx[j] - bx[i] + dy = by[j] - by[i] + dist = np.sqrt(dx * dx + dy * dy) + if dist < perception and dist > 0.01: + count += 1 + # Separation: steer away from close neighbors + if dist < perception * 0.4: + sep_x -= dx / (dist * dist) + sep_y -= dy / (dist * dist) + # Alignment: match velocity + ali_x += bvx[j] + ali_y += bvy[j] + # Cohesion: steer toward center of group + coh_x += bx[j] + coh_y += by[j] + + if count > 0: + # Normalize and weight + ax[i] += sep_x * sep_weight + ay[i] += sep_y * sep_weight + ax[i] += (ali_x / count - bvx[i]) * ali_weight * 0.1 + ay[i] += (ali_y / count - bvy[i]) * ali_weight * 0.1 + ax[i] += (coh_x / count - bx[i]) * coh_weight * 0.01 + ay[i] += (coh_y / count - by[i]) * coh_weight * 0.01 + + # Audio reactivity: bass pushes boids outward from center + if f.get("bass", 0) > 0.5: + cx_g, cy_g = g.cols / 2, g.rows / 2 + dx = bx - cx_g; dy = by - cy_g + dist = np.sqrt(dx**2 + dy**2) + 1 + ax += (dx / dist) * f["bass"] * 2 + ay += (dy / dist) * f["bass"] * 2 + + # Update velocity and position + bvx += ax; bvy += ay + # Clamp speed + speed = np.sqrt(bvx**2 + bvy**2) + 1e-10 + over = speed > max_speed + bvx[over] *= max_speed / speed[over] + bvy[over] *= max_speed / speed[over] + bx += bvx; by += bvy + + # Wrap at edges + bx %= g.cols; by %= g.rows + + S["boid_x"] = bx; S["boid_y"] = by + S["boid_vx"] = bvx; S["boid_vy"] = bvy + + # Draw + ch = np.full((g.rows, g.cols), " ", dtype="U1") + co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8) + for i in range(n): + r, c = int(by[i]) % g.rows, int(bx[i]) % g.cols + ch[r, c] = S["boid_ch"][i] + spd = min(1.0, speed[i] / max_speed) + R, G, B = hsv2rgb_scalar(spd * 0.3, 0.8, 0.5 + spd * 0.5) + co[r, c] = (R, G, B) + return ch, co +``` + +### Flow Field Particles + +Particles that follow the gradient of a value field. Any `vf_*` function becomes a "river" that carries particles: + +```python +def update_flow_particles(S, g, f, flow_field, n=500, speed=1.0, + life_drain=0.005, emit_rate=10, + char_set=None): + """Particles steered by a value field gradient. + + flow_field: float32 (rows, cols) — the field particles follow. + Particles flow from low to high values (uphill) or along + the gradient direction. + """ + if char_set is None: + char_set = list("·•∘◦°⋅") + if "fp_x" not in S: + S["fp_x"] = []; S["fp_y"] = []; S["fp_vx"] = []; S["fp_vy"] = [] + S["fp_life"] = []; S["fp_ch"] = [] + + # Emit new particles at random positions + for _ in range(emit_rate): + if len(S["fp_x"]) < n: + S["fp_x"].append(random.uniform(0, g.cols - 1)) + S["fp_y"].append(random.uniform(0, g.rows - 1)) + S["fp_vx"].append(0.0); S["fp_vy"].append(0.0) + S["fp_life"].append(1.0) + S["fp_ch"].append(random.choice(char_set)) + + # Compute gradient of flow field (central differences) + pad = np.pad(flow_field, 1, mode="wrap") + grad_x = (pad[1:-1, 2:] - pad[1:-1, :-2]) * 0.5 + grad_y = (pad[2:, 1:-1] - pad[:-2, 1:-1]) * 0.5 + + # Update particles + i = 0 + while i < len(S["fp_x"]): + px, py = S["fp_x"][i], S["fp_y"][i] + # Sample gradient at particle position + gc = int(px) % g.cols; gr = int(py) % g.rows + gx = grad_x[gr, gc]; gy = grad_y[gr, gc] + # Steer velocity toward gradient direction + S["fp_vx"][i] = S["fp_vx"][i] * 0.9 + gx * speed * 10 + S["fp_vy"][i] = S["fp_vy"][i] * 0.9 + gy * speed * 10 + S["fp_x"][i] += S["fp_vx"][i] + S["fp_y"][i] += S["fp_vy"][i] + S["fp_life"][i] -= life_drain + + if S["fp_life"][i] <= 0: + for k in ("fp_x", "fp_y", "fp_vx", "fp_vy", "fp_life", "fp_ch"): + S[k].pop(i) + else: + i += 1 + + # Draw + ch = np.full((g.rows, g.cols), " ", dtype="U1") + co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8) + for i in range(len(S["fp_x"])): + r = int(S["fp_y"][i]) % g.rows + c = int(S["fp_x"][i]) % g.cols + ch[r, c] = S["fp_ch"][i] + v = S["fp_life"][i] + co[r, c] = (int(v * 200), int(v * 180), int(v * 255)) + return ch, co +``` + +### Particle Trails + +Draw fading lines between current and previous positions: + +```python +def draw_particle_trails(S, g, trail_key="trails", max_trail=8, fade=0.7): + """Add trails to any particle system. Call after updating positions. + Stores previous positions in S[trail_key] and draws fading lines. + + Expects S to have 'px', 'py' lists (standard particle keys). + max_trail: number of previous positions to remember + fade: brightness multiplier per trail step (0.7 = 70% each step back) + """ + if trail_key not in S: + S[trail_key] = [] + + # Store current positions + current = list(zip( + [int(y) for y in S.get("py", [])], + [int(x) for x in S.get("px", [])] + )) + S[trail_key].append(current) + if len(S[trail_key]) > max_trail: + S[trail_key] = S[trail_key][-max_trail:] + + # Draw trails onto char/color arrays + ch = np.full((g.rows, g.cols), " ", dtype="U1") + co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8) + trail_chars = list("·∘◦°⋅.,'`") + + for age, positions in enumerate(reversed(S[trail_key])): + bri = fade ** age + if bri < 0.05: + break + ci = min(age, len(trail_chars) - 1) + for r, c in positions: + if 0 <= r < g.rows and 0 <= c < g.cols and ch[r, c] == " ": + ch[r, c] = trail_chars[ci] + v = int(bri * 180) + co[r, c] = (v, v, int(v * 0.8)) + return ch, co +``` + --- ## Rain / Matrix Effects ### Column Rain (Vectorized) ```python -def eff_matrix_rain(g, f, t, state, hue=0.33, bri=0.6, pal=PAL_KATA, +def eff_matrix_rain(g, f, t, S, hue=0.33, bri=0.6, pal=PAL_KATA, speed_base=0.5, speed_beat=3.0): - """Vectorized matrix rain. state dict persists column positions.""" - if "ry" not in state or len(state["ry"]) != g.cols: - state["ry"] = np.random.uniform(-g.rows, g.rows, g.cols).astype(np.float32) - state["rsp"] = np.random.uniform(0.3, 2.0, g.cols).astype(np.float32) - state["rln"] = np.random.randint(8, 40, g.cols) - state["rch"] = np.random.randint(0, len(pal), (g.rows, g.cols)) # pre-assign chars + """Vectorized matrix rain. S dict persists column positions.""" + if "ry" not in S or len(S["ry"]) != g.cols: + S["ry"] = np.random.uniform(-g.rows, g.rows, g.cols).astype(np.float32) + S["rsp"] = np.random.uniform(0.3, 2.0, g.cols).astype(np.float32) + S["rln"] = np.random.randint(8, 40, g.cols) + S["rch"] = np.random.randint(0, len(pal), (g.rows, g.cols)) # pre-assign chars speed_mult = speed_base + f.get("bass", 0.3)*speed_beat + f.get("sub_r", 0.3)*3 if f.get("beat", 0) > 0: speed_mult *= 2.5 - state["ry"] += state["rsp"] * speed_mult + S["ry"] += S["rsp"] * speed_mult # Reset columns that fall past bottom - rst = (state["ry"] - state["rln"]) > g.rows - state["ry"][rst] = np.random.uniform(-25, -2, rst.sum()) + rst = (S["ry"] - S["rln"]) > g.rows + S["ry"][rst] = np.random.uniform(-25, -2, rst.sum()) # Vectorized draw using fancy indexing ch = np.full((g.rows, g.cols), " ", dtype="U1") co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8) - heads = state["ry"].astype(int) + heads = S["ry"].astype(int) for c in range(g.cols): head = heads[c] - trail_len = state["rln"][c] + trail_len = S["rln"][c] for i in range(trail_len): row = head - i if 0 <= row < g.rows: fade = 1.0 - i / trail_len - ci = state["rch"][row, c] % len(pal) + ci = S["rch"][row, c] % len(pal) ch[row, c] = pal[ci] v = fade * bri * 255 if i == 0: # head is bright white-ish @@ -383,7 +618,7 @@ def eff_matrix_rain(g, f, t, state, hue=0.33, bri=0.6, pal=PAL_KATA, else: R, G, B = hsv2rgb_single(hue, 0.7, fade * bri) co[row, c] = (R, G, B) - return ch, co, state + return ch, co, S ``` --- @@ -666,6 +901,8 @@ class ConditionalEffect(EffectNode): These produce float32 arrays `(rows, cols)` in range [0,1]. They are the raw visual patterns. All have signature `(g, f, t, S, **params) -> float32 array`. +#### Trigonometric Fields (sine/cosine-based) + ```python def vf_sinefield(g, f, t, S, bri=0.5, freq=(0.13, 0.17, 0.07, 0.09), speed=(0.5, -0.4, -0.3, 0.2)): @@ -770,6 +1007,508 @@ def vf_noise_static(g, f, t, S, density=0.4): return np.random.random((g.rows, g.cols)).astype(np.float32) * density * (0.5 + f.get("rms",0.3)*0.5) ``` +#### Noise-Based Fields (organic, non-periodic) + +These produce qualitatively different textures from sine-based fields — organic, non-repeating, without visible axis alignment. They're the foundation of high-end generative art. + +```python +def _hash2d(ix, iy): + """Integer-coordinate hash for gradient noise. Returns float32 in [0,1].""" + # Good-quality hash via large prime mixing + n = ix * 374761393 + iy * 668265263 + n = (n ^ (n >> 13)) * 1274126177 + return ((n ^ (n >> 16)) & 0x7fffffff).astype(np.float32) / 0x7fffffff + +def _smoothstep(t): + """Hermite smoothstep: 3t^2 - 2t^3. Smooth interpolation in [0,1].""" + t = np.clip(t, 0, 1) + return t * t * (3 - 2 * t) + +def _smootherstep(t): + """Perlin's improved smoothstep: 6t^5 - 15t^4 + 10t^3. C2-continuous.""" + t = np.clip(t, 0, 1) + return t * t * t * (t * (t * 6 - 15) + 10) + +def _value_noise_2d(x, y): + """2D value noise at arbitrary float coordinates. Returns float32 in [0,1]. + x, y: float32 arrays of same shape.""" + ix = np.floor(x).astype(np.int64) + iy = np.floor(y).astype(np.int64) + fx = _smootherstep(x - ix) + fy = _smootherstep(y - iy) + # 4-corner hashes + n00 = _hash2d(ix, iy) + n10 = _hash2d(ix + 1, iy) + n01 = _hash2d(ix, iy + 1) + n11 = _hash2d(ix + 1, iy + 1) + # Bilinear interpolation + nx0 = n00 * (1 - fx) + n10 * fx + nx1 = n01 * (1 - fx) + n11 * fx + return nx0 * (1 - fy) + nx1 * fy + +def vf_noise(g, f, t, S, freq=0.08, speed=0.3, bri=0.7): + """Value noise. Smooth, organic, no axis alignment artifacts. + freq: spatial frequency (higher = finer detail). + speed: temporal scroll rate.""" + x = g.cc * freq + t * speed + y = g.rr * freq * 0.8 - t * speed * 0.4 + return np.clip(_value_noise_2d(x, y) * bri, 0, 1) + +def vf_fbm(g, f, t, S, octaves=5, freq=0.06, lacunarity=2.0, gain=0.5, + speed=0.2, bri=0.8): + """Fractal Brownian Motion — octaved noise with lacunarity/gain control. + The standard building block for clouds, terrain, smoke, organic textures. + + octaves: number of noise layers (more = finer detail, more cost) + freq: base spatial frequency + lacunarity: frequency multiplier per octave (2.0 = standard) + gain: amplitude multiplier per octave (0.5 = standard, <0.5 = smoother) + speed: temporal evolution rate + """ + val = np.zeros((g.rows, g.cols), dtype=np.float32) + amplitude = 1.0 + f_x = freq + f_y = freq * 0.85 # slight anisotropy avoids grid artifacts + for i in range(octaves): + phase = t * speed * (1 + i * 0.3) + x = g.cc * f_x + phase + i * 17.3 # offset per octave + y = g.rr * f_y - phase * 0.6 + i * 31.7 + val = val + _value_noise_2d(x, y) * amplitude + amplitude *= gain + f_x *= lacunarity + f_y *= lacunarity + # Normalize to [0,1] + max_amp = (1 - gain ** octaves) / (1 - gain) if gain != 1 else octaves + return np.clip(val / max_amp * bri * (0.6 + f.get("rms", 0.3) * 0.6), 0, 1) + +def vf_domain_warp(g, f, t, S, base_fn=None, warp_fn=None, + warp_strength=15.0, freq=0.06, speed=0.2): + """Domain warping — feed one noise field's output as coordinate offsets + into another noise field. Produces flowing, melting organic distortion. + Signature technique of high-end generative art (Inigo Quilez). + + base_fn: value field to distort (default: fbm) + warp_fn: value field for displacement (default: noise at different freq) + warp_strength: how many grid cells to displace (higher = more warped) + """ + # Warp field: displacement in x and y + wx = _value_noise_2d(g.cc * freq * 1.3 + t * speed, g.rr * freq + 7.1) + wy = _value_noise_2d(g.cc * freq + t * speed * 0.7 + 3.2, g.rr * freq * 1.1 - 11.8) + # Center warp around 0 (noise returns [0,1], shift to [-0.5, 0.5]) + wx = (wx - 0.5) * warp_strength * (0.5 + f.get("rms", 0.3) * 1.0) + wy = (wy - 0.5) * warp_strength * (0.5 + f.get("bass", 0.3) * 0.8) + # Sample base field at warped coordinates + warped_cc = g.cc + wx + warped_rr = g.rr + wy + if base_fn is not None: + # Create a temporary grid-like object with warped coords + # Simplification: evaluate base_fn with modified coordinates + val = _value_noise_2d(warped_cc * freq * 0.8 + t * speed * 0.5, + warped_rr * freq * 0.7 - t * speed * 0.3) + else: + # Default: fbm at warped coordinates + val = np.zeros((g.rows, g.cols), dtype=np.float32) + amp = 1.0 + fx, fy = freq * 0.8, freq * 0.7 + for i in range(4): + val = val + _value_noise_2d(warped_cc * fx + t * speed * 0.5 + i * 13.7, + warped_rr * fy - t * speed * 0.3 + i * 27.3) * amp + amp *= 0.5; fx *= 2.0; fy *= 2.0 + val = val / 1.875 # normalize 4-octave sum + return np.clip(val * 0.8, 0, 1) + +def vf_voronoi(g, f, t, S, n_cells=20, speed=0.3, edge_width=1.5, + mode="distance", seed=42): + """Voronoi diagram as value field. Proper implementation with + nearest/second-nearest distance for cell interiors and edges. + + mode: "distance" (bright at center, dark at edges), + "edge" (bright at cell boundaries), + "cell_id" (flat color per cell — use with discrete palette) + edge_width: thickness of edge highlight (for "edge" mode) + """ + rng = np.random.RandomState(seed) + # Animated cell centers + cx = rng.rand(n_cells).astype(np.float32) * g.cols + cy = rng.rand(n_cells).astype(np.float32) * g.rows + vx = (rng.rand(n_cells).astype(np.float32) - 0.5) * speed * 10 + vy = (rng.rand(n_cells).astype(np.float32) - 0.5) * speed * 10 + cx_t = (cx + vx * np.sin(t * 0.5 + np.arange(n_cells) * 0.8)) % g.cols + cy_t = (cy + vy * np.cos(t * 0.4 + np.arange(n_cells) * 1.1)) % g.rows + + # Compute nearest and second-nearest distance + d1 = np.full((g.rows, g.cols), 1e9, dtype=np.float32) + d2 = np.full((g.rows, g.cols), 1e9, dtype=np.float32) + id1 = np.zeros((g.rows, g.cols), dtype=np.int32) + for i in range(n_cells): + d = np.sqrt((g.cc - cx_t[i]) ** 2 + (g.rr - cy_t[i]) ** 2) + mask = d < d1 + d2 = np.where(mask, d1, np.minimum(d2, d)) + id1 = np.where(mask, i, id1) + d1 = np.minimum(d1, d) + + if mode == "edge": + # Edges: where d2 - d1 is small + edge_val = np.clip(1.0 - (d2 - d1) / edge_width, 0, 1) + return edge_val * (0.5 + f.get("rms", 0.3) * 0.8) + elif mode == "cell_id": + # Flat per-cell value + return (id1.astype(np.float32) / n_cells) % 1.0 + else: + # Distance: bright near center, dark at edges + max_d = g.cols * 0.15 + return np.clip(1.0 - d1 / max_d, 0, 1) * (0.5 + f.get("rms", 0.3) * 0.7) +``` + +#### Simulation-Based Fields (emergent, evolving) + +These use persistent state `S` to evolve patterns frame-by-frame. They produce complexity that can't be achieved with stateless math. + +```python +def vf_reaction_diffusion(g, f, t, S, feed=0.055, kill=0.062, + da=1.0, db=0.5, dt=1.0, steps_per_frame=8, + init_mode="spots"): + """Gray-Scott reaction-diffusion model. Produces coral, leopard spots, + mitosis, worm-like, and labyrinthine patterns depending on feed/kill. + + The two chemicals A and B interact: + A + 2B → 3B (autocatalytic) + B → P (decay) + feed: rate A is replenished, kill: rate B decays + Different feed/kill ratios produce radically different patterns. + + Presets (feed, kill): + Spots/dots: (0.055, 0.062) + Worms/stripes: (0.046, 0.063) + Coral/branching: (0.037, 0.060) + Mitosis/splitting: (0.028, 0.062) + Labyrinth/maze: (0.029, 0.057) + Holes/negative: (0.039, 0.058) + Chaos/unstable: (0.026, 0.051) + + steps_per_frame: simulation steps per video frame (more = faster evolution) + """ + key = "rd_" + str(id(g)) # unique per grid + if key + "_a" not in S: + # Initialize chemical fields + A = np.ones((g.rows, g.cols), dtype=np.float32) + B = np.zeros((g.rows, g.cols), dtype=np.float32) + if init_mode == "spots": + # Random seed spots + rng = np.random.RandomState(42) + for _ in range(max(3, g.rows * g.cols // 200)): + r, c = rng.randint(2, g.rows - 2), rng.randint(2, g.cols - 2) + B[r - 1:r + 2, c - 1:c + 2] = 1.0 + elif init_mode == "center": + cr, cc = g.rows // 2, g.cols // 2 + B[cr - 3:cr + 3, cc - 3:cc + 3] = 1.0 + elif init_mode == "ring": + mask = (g.dist_n > 0.2) & (g.dist_n < 0.3) + B[mask] = 1.0 + S[key + "_a"] = A + S[key + "_b"] = B + + A = S[key + "_a"] + B = S[key + "_b"] + + # Audio modulation: feed/kill shift subtly with audio + f_mod = feed + f.get("bass", 0.3) * 0.003 + k_mod = kill + f.get("hi_r", 0.3) * 0.002 + + for _ in range(steps_per_frame): + # Laplacian via 3x3 convolution kernel + # [0.05, 0.2, 0.05] + # [0.2, -1.0, 0.2] + # [0.05, 0.2, 0.05] + pA = np.pad(A, 1, mode="wrap") + pB = np.pad(B, 1, mode="wrap") + lapA = (pA[:-2, 1:-1] + pA[2:, 1:-1] + pA[1:-1, :-2] + pA[1:-1, 2:]) * 0.2 \ + + (pA[:-2, :-2] + pA[:-2, 2:] + pA[2:, :-2] + pA[2:, 2:]) * 0.05 \ + - A * 1.0 + lapB = (pB[:-2, 1:-1] + pB[2:, 1:-1] + pB[1:-1, :-2] + pB[1:-1, 2:]) * 0.2 \ + + (pB[:-2, :-2] + pB[:-2, 2:] + pB[2:, :-2] + pB[2:, 2:]) * 0.05 \ + - B * 1.0 + ABB = A * B * B + A = A + (da * lapA - ABB + f_mod * (1 - A)) * dt + B = B + (db * lapB + ABB - (f_mod + k_mod) * B) * dt + A = np.clip(A, 0, 1) + B = np.clip(B, 0, 1) + + S[key + "_a"] = A + S[key + "_b"] = B + # Output B chemical as value (the visible pattern) + return np.clip(B * 2.0, 0, 1) + +def vf_game_of_life(g, f, t, S, rule="life", birth=None, survive=None, + steps_per_frame=1, density=0.3, fade=0.92, seed=42): + """Cellular automaton as value field with analog fade trails. + Grid cells are born/die by neighbor count rules. Dead cells fade + gradually instead of snapping to black, producing ghost trails. + + rule presets: + "life": B3/S23 (Conway's Game of Life) + "coral": B3/S45678 (slow crystalline growth) + "maze": B3/S12345 (fills to labyrinth) + "anneal": B4678/S35678 (smooth blobs) + "day_night": B3678/S34678 (balanced growth/decay) + Or specify birth/survive directly as sets: birth={3}, survive={2,3} + + fade: how fast dead cells dim (0.9 = slow trails, 0.5 = fast) + """ + presets = { + "life": ({3}, {2, 3}), + "coral": ({3}, {4, 5, 6, 7, 8}), + "maze": ({3}, {1, 2, 3, 4, 5}), + "anneal": ({4, 6, 7, 8}, {3, 5, 6, 7, 8}), + "day_night": ({3, 6, 7, 8}, {3, 4, 6, 7, 8}), + } + if birth is None or survive is None: + birth, survive = presets.get(rule, presets["life"]) + + key = "gol_" + str(id(g)) + if key + "_grid" not in S: + rng = np.random.RandomState(seed) + S[key + "_grid"] = (rng.random((g.rows, g.cols)) < density).astype(np.float32) + S[key + "_display"] = S[key + "_grid"].copy() + + grid = S[key + "_grid"] + display = S[key + "_display"] + + # Beat can inject random noise + if f.get("beat", 0) > 0.5: + inject = np.random.random((g.rows, g.cols)) < 0.02 + grid = np.clip(grid + inject.astype(np.float32), 0, 1) + + for _ in range(steps_per_frame): + # Count neighbors (toroidal wrap) + padded = np.pad(grid > 0.5, 1, mode="wrap").astype(np.int8) + neighbors = (padded[:-2, :-2] + padded[:-2, 1:-1] + padded[:-2, 2:] + + padded[1:-1, :-2] + padded[1:-1, 2:] + + padded[2:, :-2] + padded[2:, 1:-1] + padded[2:, 2:]) + alive = grid > 0.5 + new_alive = np.zeros_like(grid, dtype=bool) + for b in birth: + new_alive |= (~alive) & (neighbors == b) + for s in survive: + new_alive |= alive & (neighbors == s) + grid = new_alive.astype(np.float32) + + # Analog display: alive cells = 1.0, dead cells fade + display = np.where(grid > 0.5, 1.0, display * fade) + S[key + "_grid"] = grid + S[key + "_display"] = display + return np.clip(display, 0, 1) + +def vf_strange_attractor(g, f, t, S, attractor="clifford", + n_points=50000, warmup=500, bri=0.8, seed=42, + params=None): + """Strange attractor projected to 2D density field. + Iterates N points through attractor equations, bins to grid, + produces a density map. Elegant, non-repeating curves. + + attractor presets: + "clifford": sin(a*y) + c*cos(a*x), sin(b*x) + d*cos(b*y) + "de_jong": sin(a*y) - cos(b*x), sin(c*x) - cos(d*y) + "bedhead": sin(x*y/b) + cos(a*x - y), x*sin(a*y) + cos(b*x - y) + + params: (a, b, c, d) floats — each attractor has different sweet spots. + If None, uses time-varying defaults for animation. + """ + key = "attr_" + attractor + if params is None: + # Time-varying parameters for slow morphing + a = -1.4 + np.sin(t * 0.05) * 0.3 + b = 1.6 + np.cos(t * 0.07) * 0.2 + c = 1.0 + np.sin(t * 0.03 + 1) * 0.3 + d = 0.7 + np.cos(t * 0.04 + 2) * 0.2 + else: + a, b, c, d = params + + # Iterate attractor + rng = np.random.RandomState(seed) + x = rng.uniform(-0.1, 0.1, n_points).astype(np.float64) + y = rng.uniform(-0.1, 0.1, n_points).astype(np.float64) + + # Warmup iterations (reach the attractor) + for _ in range(warmup): + if attractor == "clifford": + xn = np.sin(a * y) + c * np.cos(a * x) + yn = np.sin(b * x) + d * np.cos(b * y) + elif attractor == "de_jong": + xn = np.sin(a * y) - np.cos(b * x) + yn = np.sin(c * x) - np.cos(d * y) + elif attractor == "bedhead": + xn = np.sin(x * y / b) + np.cos(a * x - y) + yn = x * np.sin(a * y) + np.cos(b * x - y) + else: + xn = np.sin(a * y) + c * np.cos(a * x) + yn = np.sin(b * x) + d * np.cos(b * y) + x, y = xn, yn + + # Bin to grid + # Find bounds + margin = 0.1 + x_min, x_max = x.min() - margin, x.max() + margin + y_min, y_max = y.min() - margin, y.max() + margin + + # Map to grid coordinates + gx = ((x - x_min) / (x_max - x_min) * (g.cols - 1)).astype(np.int32) + gy = ((y - y_min) / (y_max - y_min) * (g.rows - 1)).astype(np.int32) + valid = (gx >= 0) & (gx < g.cols) & (gy >= 0) & (gy < g.rows) + gx, gy = gx[valid], gy[valid] + + # Accumulate density + density = np.zeros((g.rows, g.cols), dtype=np.float32) + np.add.at(density, (gy, gx), 1.0) + + # Log-scale density for visibility (most bins have few hits) + density = np.log1p(density) + mx = density.max() + if mx > 0: + density = density / mx + return np.clip(density * bri * (0.5 + f.get("rms", 0.3) * 0.8), 0, 1) +``` + +#### SDF-Based Fields (geometric precision) + +Signed Distance Fields produce mathematically precise shapes. Unlike sine fields (organic, blurry), SDFs give hard geometric boundaries with controllable edge softness. Combined with domain warping, they create "melting geometry" effects. + +All SDF primitives return a **signed distance** (negative inside, positive outside). Convert to a value field with `sdf_render()`. + +```python +def sdf_render(dist, edge_width=1.5, invert=False): + """Convert signed distance to value field [0,1]. + edge_width: controls anti-aliasing / softness of the boundary. + invert: True = bright inside shape, False = bright outside.""" + val = 1.0 - np.clip(dist / edge_width, 0, 1) if not invert else np.clip(dist / edge_width, 0, 1) + return np.clip(val, 0, 1) + +def sdf_glow(dist, falloff=0.05): + """Render SDF as glowing outline — bright at boundary, fading both directions.""" + return np.clip(np.exp(-np.abs(dist) * falloff), 0, 1) + +# --- Primitives --- + +def sdf_circle(g, cx_frac=0.5, cy_frac=0.5, radius=0.3): + """Circle SDF. cx/cy/radius in normalized [0,1] coordinates.""" + dx = (g.cc / g.cols - cx_frac) * (g.cols / g.rows) # aspect correction + dy = g.rr / g.rows - cy_frac + return np.sqrt(dx**2 + dy**2) - radius + +def sdf_box(g, cx_frac=0.5, cy_frac=0.5, w=0.3, h=0.2, round_r=0.0): + """Rounded rectangle SDF.""" + dx = np.abs(g.cc / g.cols - cx_frac) * (g.cols / g.rows) - w + round_r + dy = np.abs(g.rr / g.rows - cy_frac) - h + round_r + outside = np.sqrt(np.maximum(dx, 0)**2 + np.maximum(dy, 0)**2) + inside = np.minimum(np.maximum(dx, dy), 0) + return outside + inside - round_r + +def sdf_ring(g, cx_frac=0.5, cy_frac=0.5, radius=0.3, thickness=0.03): + """Ring (annulus) SDF.""" + d = sdf_circle(g, cx_frac, cy_frac, radius) + return np.abs(d) - thickness + +def sdf_line(g, x0=0.2, y0=0.5, x1=0.8, y1=0.5, thickness=0.01): + """Line segment SDF between two points (normalized coords).""" + ax = g.cc / g.cols * (g.cols / g.rows) - x0 * (g.cols / g.rows) + ay = g.rr / g.rows - y0 + bx = (x1 - x0) * (g.cols / g.rows) + by = y1 - y0 + h = np.clip((ax * bx + ay * by) / (bx * bx + by * by + 1e-10), 0, 1) + dx = ax - bx * h + dy = ay - by * h + return np.sqrt(dx**2 + dy**2) - thickness + +def sdf_triangle(g, cx=0.5, cy=0.5, size=0.25): + """Equilateral triangle SDF centered at (cx, cy).""" + px = (g.cc / g.cols - cx) * (g.cols / g.rows) / size + py = (g.rr / g.rows - cy) / size + # Equilateral triangle math + k = np.sqrt(3.0) + px = np.abs(px) - 1.0 + py = py + 1.0 / k + cond = px + k * py > 0 + px2 = np.where(cond, (px - k * py) / 2.0, px) + py2 = np.where(cond, (-k * px - py) / 2.0, py) + px2 = np.clip(px2, -2.0, 0.0) + return -np.sqrt(px2**2 + py2**2) * np.sign(py2) * size + +def sdf_star(g, cx=0.5, cy=0.5, n_points=5, outer_r=0.25, inner_r=0.12): + """Star polygon SDF — n-pointed star.""" + px = (g.cc / g.cols - cx) * (g.cols / g.rows) + py = g.rr / g.rows - cy + angle = np.arctan2(py, px) + dist = np.sqrt(px**2 + py**2) + # Modular angle for star symmetry + wedge = 2 * np.pi / n_points + a = np.abs((angle % wedge) - wedge / 2) + # Interpolate radius between inner and outer + r_at_angle = inner_r + (outer_r - inner_r) * np.clip(np.cos(a * n_points) * 0.5 + 0.5, 0, 1) + return dist - r_at_angle + +def sdf_heart(g, cx=0.5, cy=0.45, size=0.25): + """Heart shape SDF.""" + px = (g.cc / g.cols - cx) * (g.cols / g.rows) / size + py = -(g.rr / g.rows - cy) / size + 0.3 # flip y, offset + px = np.abs(px) + cond = (px + py) > 1.0 + d1 = np.sqrt((px - 0.25)**2 + (py - 0.75)**2) - np.sqrt(2.0) / 4.0 + d2 = np.sqrt((px + py - 1.0)**2) / np.sqrt(2.0) + return np.where(cond, d1, d2) * size + +# --- Combinators --- + +def sdf_union(d1, d2): + """Boolean union — shape is wherever either SDF is inside.""" + return np.minimum(d1, d2) + +def sdf_intersect(d1, d2): + """Boolean intersection — shape is where both SDFs overlap.""" + return np.maximum(d1, d2) + +def sdf_subtract(d1, d2): + """Boolean subtraction — d1 minus d2.""" + return np.maximum(d1, -d2) + +def sdf_smooth_union(d1, d2, k=0.1): + """Smooth minimum (polynomial) — blends shapes with rounded join. + k: smoothing radius. Higher = more rounding.""" + h = np.clip(0.5 + 0.5 * (d2 - d1) / k, 0, 1) + return d2 * (1 - h) + d1 * h - k * h * (1 - h) + +def sdf_smooth_subtract(d1, d2, k=0.1): + """Smooth subtraction — d1 minus d2 with rounded edge.""" + return sdf_smooth_union(d1, -d2, k) + +def sdf_repeat(g, sdf_fn, spacing_x=0.25, spacing_y=0.25, **sdf_kwargs): + """Tile an SDF primitive infinitely. spacing in normalized coords.""" + # Modular coordinates + mod_cc = (g.cc / g.cols) % spacing_x - spacing_x / 2 + mod_rr = (g.rr / g.rows) % spacing_y - spacing_y / 2 + # Create modified grid-like arrays for the SDF + # This is a simplified approach — build a temporary namespace + class ModGrid: + pass + mg = ModGrid() + mg.cc = mod_cc * g.cols; mg.rr = mod_rr * g.rows + mg.cols = g.cols; mg.rows = g.rows + return sdf_fn(mg, **sdf_kwargs) + +# --- SDF as Value Field --- + +def vf_sdf(g, f, t, S, sdf_fn=sdf_circle, edge_width=1.5, glow=False, + glow_falloff=0.03, animate=True, **sdf_kwargs): + """Wrap any SDF primitive as a standard vf_* value field. + If animate=True, applies slow rotation and breathing to the shape.""" + if animate: + sdf_kwargs.setdefault("cx_frac", 0.5) + sdf_kwargs.setdefault("cy_frac", 0.5) + d = sdf_fn(g, **sdf_kwargs) + if glow: + return sdf_glow(d, glow_falloff) * (0.5 + f.get("rms", 0.3) * 0.8) + return sdf_render(d, edge_width) * (0.5 + f.get("rms", 0.3) * 0.8) +``` + ### Hue Field Generators (Color Mapping) These produce float32 hue arrays [0,1]. Independently combinable with any value field. Each is a factory returning a closure with signature `(g, f, t, S) -> float32 array`. Can also be a plain float for fixed hue. @@ -832,6 +1571,343 @@ def hf_plasma(speed=0.3): return fn ``` +--- + +## Coordinate Transforms + +UV-space transforms applied **before** effect evaluation. Any `vf_*` function can be rotated, zoomed, tiled, or distorted by transforming the grid coordinates it sees. + +### Transform Helpers + +```python +def uv_rotate(g, angle): + """Rotate UV coordinates around grid center. + Returns (rotated_cc, rotated_rr) arrays — use in place of g.cc, g.rr.""" + cx, cy = g.cols / 2.0, g.rows / 2.0 + cos_a, sin_a = np.cos(angle), np.sin(angle) + dx = g.cc - cx + dy = g.rr - cy + return cx + dx * cos_a - dy * sin_a, cy + dx * sin_a + dy * cos_a + +def uv_scale(g, sx=1.0, sy=1.0, cx_frac=0.5, cy_frac=0.5): + """Scale UV coordinates around a center point. + sx, sy > 1 = zoom in (fewer repeats), < 1 = zoom out (more repeats).""" + cx = g.cols * cx_frac; cy = g.rows * cy_frac + return cx + (g.cc - cx) / sx, cy + (g.rr - cy) / sy + +def uv_skew(g, kx=0.0, ky=0.0): + """Skew UV coordinates. kx shears horizontally, ky vertically.""" + return g.cc + g.rr * kx, g.rr + g.cc * ky + +def uv_tile(g, nx=3.0, ny=3.0, mirror=False): + """Tile UV coordinates. nx, ny = number of repeats. + mirror=True: alternating tiles are flipped (seamless).""" + u = (g.cc / g.cols * nx) % 1.0 + v = (g.rr / g.rows * ny) % 1.0 + if mirror: + flip_u = ((g.cc / g.cols * nx).astype(int) % 2) == 1 + flip_v = ((g.rr / g.rows * ny).astype(int) % 2) == 1 + u = np.where(flip_u, 1.0 - u, u) + v = np.where(flip_v, 1.0 - v, v) + return u * g.cols, v * g.rows + +def uv_polar(g): + """Convert Cartesian to polar UV. Returns (angle_as_cc, dist_as_rr). + Use to make any linear effect radial.""" + # Angle wraps [0, cols), distance wraps [0, rows) + return g.angle / (2 * np.pi) * g.cols, g.dist_n * g.rows + +def uv_cartesian_from_polar(g): + """Convert polar-addressed effects back to Cartesian. + Treats g.cc as angle and g.rr as radius.""" + angle = g.cc / g.cols * 2 * np.pi + radius = g.rr / g.rows + cx, cy = g.cols / 2.0, g.rows / 2.0 + return cx + radius * np.cos(angle) * cx, cy + radius * np.sin(angle) * cy + +def uv_twist(g, amount=2.0): + """Twist: rotation increases with distance from center. Creates spiral distortion.""" + twist_angle = g.dist_n * amount + return uv_rotate_raw(g.cc, g.rr, g.cols / 2, g.rows / 2, twist_angle) + +def uv_rotate_raw(cc, rr, cx, cy, angle): + """Raw rotation on arbitrary coordinate arrays.""" + cos_a, sin_a = np.cos(angle), np.sin(angle) + dx = cc - cx; dy = rr - cy + return cx + dx * cos_a - dy * sin_a, cy + dx * sin_a + dy * cos_a + +def uv_fisheye(g, strength=1.5): + """Fisheye / barrel distortion on UV coordinates.""" + cx, cy = g.cols / 2.0, g.rows / 2.0 + dx = (g.cc - cx) / cx + dy = (g.rr - cy) / cy + r = np.sqrt(dx**2 + dy**2) + r_distort = np.power(r, strength) + scale = np.where(r > 0, r_distort / (r + 1e-10), 1.0) + return cx + dx * scale * cx, cy + dy * scale * cy + +def uv_wave(g, t, freq=0.1, amp=3.0, axis="x"): + """Sinusoidal coordinate displacement. Wobbles the UV space.""" + if axis == "x": + return g.cc + np.sin(g.rr * freq + t * 3) * amp, g.rr + else: + return g.cc, g.rr + np.sin(g.cc * freq + t * 3) * amp + +def uv_mobius(g, a=1.0, b=0.0, c=0.0, d=1.0): + """Möbius transformation (conformal map): f(z) = (az + b) / (cz + d). + Operates on complex plane. Produces mathematically precise, visually + striking inversions and circular transforms.""" + cx, cy = g.cols / 2.0, g.rows / 2.0 + # Map grid to complex plane [-1, 1] + zr = (g.cc - cx) / cx + zi = (g.rr - cy) / cy + # Complex division: (a*z + b) / (c*z + d) + num_r = a * zr - 0 * zi + b # imaginary parts of a,b,c,d = 0 for real params + num_i = a * zi + 0 * zr + 0 + den_r = c * zr - 0 * zi + d + den_i = c * zi + 0 * zr + 0 + denom = den_r**2 + den_i**2 + 1e-10 + wr = (num_r * den_r + num_i * den_i) / denom + wi = (num_i * den_r - num_r * den_i) / denom + return cx + wr * cx, cy + wi * cy +``` + +### Using Transforms with Value Fields + +Transforms modify what coordinates a value field sees. Wrap the transform around the `vf_*` call: + +```python +# Rotate a plasma field 45 degrees +def vf_rotated_plasma(g, f, t, S): + rc, rr = uv_rotate(g, np.pi / 4 + t * 0.1) + class TG: # transformed grid + pass + tg = TG(); tg.cc = rc; tg.rr = rr + tg.rows = g.rows; tg.cols = g.cols + tg.dist_n = g.dist_n; tg.angle = g.angle; tg.dist = g.dist + return vf_plasma(tg, f, t, S) + +# Tile a vortex 3x3 with mirror +def vf_tiled_vortex(g, f, t, S): + tc, tr = uv_tile(g, 3, 3, mirror=True) + class TG: + pass + tg = TG(); tg.cc = tc; tg.rr = tr + tg.rows = g.rows; tg.cols = g.cols + tg.dist = np.sqrt((tc - g.cols/2)**2 + (tr - g.rows/2)**2) + tg.dist_n = tg.dist / (tg.dist.max() + 1e-10) + tg.angle = np.arctan2(tr - g.rows/2, tc - g.cols/2) + return vf_vortex(tg, f, t, S) + +# Helper: create transformed grid from coordinate arrays +def make_tgrid(g, new_cc, new_rr): + """Build a grid-like object with transformed coordinates. + Preserves rows/cols for sizing, recomputes polar coords.""" + class TG: + pass + tg = TG() + tg.cc = new_cc; tg.rr = new_rr + tg.rows = g.rows; tg.cols = g.cols + cx, cy = g.cols / 2.0, g.rows / 2.0 + dx = new_cc - cx; dy = new_rr - cy + tg.dist = np.sqrt(dx**2 + dy**2) + tg.dist_n = tg.dist / (max(cx, cy) + 1e-10) + tg.angle = np.arctan2(dy, dx) + tg.dx = dx; tg.dy = dy + tg.dx_n = dx / max(g.cols, 1) + tg.dy_n = dy / max(g.rows, 1) + return tg +``` + +--- + +## Temporal Coherence + +Tools for smooth, intentional parameter evolution over time. Replaces the default pattern of either static parameters or raw audio reactivity. + +### Easing Functions + +Standard animation easing curves. All take `t` in [0,1] and return [0,1]: + +```python +def ease_linear(t): return t +def ease_in_quad(t): return t * t +def ease_out_quad(t): return t * (2 - t) +def ease_in_out_quad(t): return np.where(t < 0.5, 2*t*t, -1 + (4-2*t)*t) +def ease_in_cubic(t): return t**3 +def ease_out_cubic(t): return (t - 1)**3 + 1 +def ease_in_out_cubic(t): + return np.where(t < 0.5, 4*t**3, 1 - (-2*t + 2)**3 / 2) +def ease_in_expo(t): return np.where(t == 0, 0, 2**(10*(t-1))) +def ease_out_expo(t): return np.where(t == 1, 1, 1 - 2**(-10*t)) +def ease_elastic(t): + """Elastic ease-out — overshoots then settles.""" + return np.where(t == 0, 0, np.where(t == 1, 1, + 2**(-10*t) * np.sin((t*10 - 0.75) * (2*np.pi) / 3) + 1)) +def ease_bounce(t): + """Bounce ease-out — bounces at the end.""" + t = np.asarray(t, dtype=np.float64) + result = np.empty_like(t) + m1 = t < 1/2.75 + m2 = (~m1) & (t < 2/2.75) + m3 = (~m1) & (~m2) & (t < 2.5/2.75) + m4 = ~(m1 | m2 | m3) + result[m1] = 7.5625 * t[m1]**2 + t2 = t[m2] - 1.5/2.75; result[m2] = 7.5625 * t2**2 + 0.75 + t3 = t[m3] - 2.25/2.75; result[m3] = 7.5625 * t3**2 + 0.9375 + t4 = t[m4] - 2.625/2.75; result[m4] = 7.5625 * t4**2 + 0.984375 + return result +``` + +### Keyframe Interpolation + +Define parameter values at specific times. Interpolates between them with easing: + +```python +def keyframe(t, points, ease_fn=ease_in_out_cubic, loop=False): + """Interpolate between keyframed values. + + Args: + t: current time (float, seconds) + points: list of (time, value) tuples, sorted by time + ease_fn: easing function for interpolation + loop: if True, wraps around after last keyframe + + Returns: + interpolated value at time t + + Example: + twist = keyframe(t, [(0, 1.0), (5, 6.0), (10, 2.0)], ease_out_cubic) + """ + if not points: + return 0.0 + if loop: + period = points[-1][0] - points[0][0] + if period > 0: + t = points[0][0] + (t - points[0][0]) % period + + # Clamp to range + if t <= points[0][0]: + return points[0][1] + if t >= points[-1][0]: + return points[-1][1] + + # Find surrounding keyframes + for i in range(len(points) - 1): + t0, v0 = points[i] + t1, v1 = points[i + 1] + if t0 <= t <= t1: + progress = (t - t0) / (t1 - t0) + eased = ease_fn(progress) + return v0 + (v1 - v0) * eased + + return points[-1][1] + +def keyframe_array(t, points, ease_fn=ease_in_out_cubic): + """Keyframe interpolation that works with numpy arrays as values. + points: list of (time, np.array) tuples.""" + if t <= points[0][0]: return points[0][1].copy() + if t >= points[-1][0]: return points[-1][1].copy() + for i in range(len(points) - 1): + t0, v0 = points[i] + t1, v1 = points[i + 1] + if t0 <= t <= t1: + progress = ease_fn((t - t0) / (t1 - t0)) + return v0 * (1 - progress) + v1 * progress + return points[-1][1].copy() +``` + +### Value Field Morphing + +Smooth transition between two different value fields: + +```python +def vf_morph(g, f, t, S, vf_a, vf_b, t_start, t_end, + ease_fn=ease_in_out_cubic): + """Morph between two value fields over a time range. + + Usage: + val = vf_morph(g, f, t, S, + lambda g,f,t,S: vf_plasma(g,f,t,S), + lambda g,f,t,S: vf_vortex(g,f,t,S, twist=5), + t_start=10.0, t_end=15.0) + """ + if t <= t_start: + return vf_a(g, f, t, S) + if t >= t_end: + return vf_b(g, f, t, S) + progress = ease_fn((t - t_start) / (t_end - t_start)) + a = vf_a(g, f, t, S) + b = vf_b(g, f, t, S) + return a * (1 - progress) + b * progress + +def vf_sequence(g, f, t, S, fields, durations, crossfade=1.0, + ease_fn=ease_in_out_cubic): + """Cycle through a sequence of value fields with crossfades. + + fields: list of vf_* callables + durations: list of float seconds per field + crossfade: seconds of overlap between adjacent fields + """ + total = sum(durations) + t_local = t % total # loop + elapsed = 0 + for i, dur in enumerate(durations): + if t_local < elapsed + dur: + # Current field + base = fields[i](g, f, t, S) + # Check if we're in a crossfade zone + time_in = t_local - elapsed + time_left = dur - time_in + if time_in < crossfade and i > 0: + # Fading in from previous + prev = fields[(i - 1) % len(fields)](g, f, t, S) + blend = ease_fn(time_in / crossfade) + return prev * (1 - blend) + base * blend + if time_left < crossfade and i < len(fields) - 1: + # Fading out to next + nxt = fields[(i + 1) % len(fields)](g, f, t, S) + blend = ease_fn(1 - time_left / crossfade) + return base * (1 - blend) + nxt * blend + return base + elapsed += dur + return fields[-1](g, f, t, S) +``` + +### Temporal Noise + +3D noise sampled at `(x, y, t)` — patterns evolve smoothly in time without per-frame discontinuities: + +```python +def vf_temporal_noise(g, f, t, S, freq=0.06, t_freq=0.3, octaves=4, + bri=0.8): + """Noise field that evolves smoothly in time. Uses 3D noise via + two 2D noise lookups combined with temporal interpolation. + + Unlike vf_fbm which scrolls noise (creating directional motion), + this morphs the pattern in-place — cells brighten and dim without + the field moving in any direction.""" + # Two noise samples at floor/ceil of temporal coordinate + t_scaled = t * t_freq + t_lo = np.floor(t_scaled) + t_frac = _smootherstep(np.full((g.rows, g.cols), t_scaled - t_lo, dtype=np.float32)) + + val_lo = np.zeros((g.rows, g.cols), dtype=np.float32) + val_hi = np.zeros((g.rows, g.cols), dtype=np.float32) + amp = 1.0; fx = freq + for i in range(octaves): + val_lo = val_lo + _value_noise_2d( + g.cc * fx + t_lo * 7.3 + i * 13, g.rr * fx + t_lo * 3.1 + i * 29) * amp + val_hi = val_hi + _value_noise_2d( + g.cc * fx + (t_lo + 1) * 7.3 + i * 13, g.rr * fx + (t_lo + 1) * 3.1 + i * 29) * amp + amp *= 0.5; fx *= 2.0 + max_amp = (1 - 0.5 ** octaves) / 0.5 + val = (val_lo * (1 - t_frac) + val_hi * t_frac) / max_amp + return np.clip(val * bri * (0.6 + f.get("rms", 0.3) * 0.6), 0, 1) +``` + +--- + ### Combining Value Fields The combinatorial explosion comes from mixing value fields with math: diff --git a/skills/creative/ascii-video/references/examples.md b/skills/creative/ascii-video/references/examples.md new file mode 100644 index 000000000..7ece489d9 --- /dev/null +++ b/skills/creative/ascii-video/references/examples.md @@ -0,0 +1,416 @@ +# Scene Examples + +**Cross-references:** +- Grid system, palettes, color (HSV + OKLAB): `architecture.md` +- Effect building blocks (value fields, noise, SDFs, particles): `effects.md` +- `_render_vf()`, blend modes, tonemap, masking: `composition.md` +- Scene protocol, render_clip, SCENES table: `scenes.md` +- Shader pipeline, feedback buffer, ShaderChain: `shaders.md` +- Input sources (audio features, video features): `inputs.md` +- Performance tuning: `optimization.md` +- Common bugs: `troubleshooting.md` + +Copy-paste-ready scene functions at increasing complexity. Each is a complete, working v2 scene function that returns a pixel canvas. See `scenes.md` for the scene protocol and `composition.md` for blend modes and tonemap. + +--- + +## Minimal — Single Grid, Single Effect + +### Breathing Plasma + +One grid, one value field, one hue field. The simplest possible scene. + +```python +def fx_breathing_plasma(r, f, t, S): + """Plasma field with time-cycling hue. Audio modulates brightness.""" + canvas = _render_vf(r, "md", + lambda g, f, t, S: vf_plasma(g, f, t, S) * 1.3, + hf_time_cycle(0.08), PAL_DENSE, f, t, S, sat=0.8) + return canvas +``` + +### Reaction-Diffusion Coral + +Single grid, simulation-based field. Evolves organically over time. + +```python +def fx_coral(r, f, t, S): + """Gray-Scott reaction-diffusion — coral branching pattern. + Slow-evolving, organic. Best for ambient/chill sections.""" + canvas = _render_vf(r, "sm", + lambda g, f, t, S: vf_reaction_diffusion(g, f, t, S, + feed=0.037, kill=0.060, steps_per_frame=6, init_mode="center"), + hf_distance(0.55, 0.015), PAL_DOTS, f, t, S, sat=0.7) + return canvas +``` + +### SDF Geometry + +Geometric shapes from SDFs. Clean, precise, graphic. + +```python +def fx_sdf_rings(r, f, t, S): + """Concentric SDF rings with smooth pulsing.""" + def val_fn(g, f, t, S): + d1 = sdf_ring(g, radius=0.15 + f.get("bass", 0.3) * 0.05, thickness=0.015) + d2 = sdf_ring(g, radius=0.25 + f.get("mid", 0.3) * 0.05, thickness=0.012) + d3 = sdf_ring(g, radius=0.35 + f.get("hi", 0.3) * 0.04, thickness=0.010) + combined = sdf_smooth_union(sdf_smooth_union(d1, d2, 0.05), d3, 0.05) + return sdf_glow(combined, falloff=0.08) * (0.5 + f.get("rms", 0.3) * 0.8) + canvas = _render_vf(r, "md", val_fn, hf_angle(0.0), PAL_STARS, f, t, S, sat=0.85) + return canvas +``` + +--- + +## Standard — Two Grids + Blend + +### Tunnel Through Noise + +Two grids at different densities, screen blended. The fine noise texture shows through the coarser tunnel characters. + +```python +def fx_tunnel_noise(r, f, t, S): + """Tunnel depth on md grid + fBM noise on sm grid, screen blended.""" + canvas_a = _render_vf(r, "md", + lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=4.0, complexity=8) * 1.2, + hf_distance(0.5, 0.02), PAL_BLOCKS, f, t, S, sat=0.7) + + canvas_b = _render_vf(r, "sm", + lambda g, f, t, S: vf_fbm(g, f, t, S, octaves=4, freq=0.05, speed=0.15) * 1.3, + hf_time_cycle(0.06), PAL_RUNE, f, t, S, sat=0.6) + + return blend_canvas(canvas_a, canvas_b, "screen", 0.7) +``` + +### Voronoi Cells + Spiral Overlay + +Voronoi cell edges with a spiral arm pattern overlaid. + +```python +def fx_voronoi_spiral(r, f, t, S): + """Voronoi edge detection on md + logarithmic spiral on lg.""" + canvas_a = _render_vf(r, "md", + lambda g, f, t, S: vf_voronoi(g, f, t, S, + n_cells=15, mode="edge", edge_width=2.0, speed=0.4), + hf_angle(0.2), PAL_CIRCUIT, f, t, S, sat=0.75) + + canvas_b = _render_vf(r, "lg", + lambda g, f, t, S: vf_spiral(g, f, t, S, n_arms=4, tightness=3.0) * 1.2, + hf_distance(0.1, 0.03), PAL_BLOCKS, f, t, S, sat=0.9) + + return blend_canvas(canvas_a, canvas_b, "exclusion", 0.6) +``` + +### Domain-Warped fBM + +Two layers of the same fBM, one domain-warped, difference-blended for psychedelic organic texture. + +```python +def fx_organic_warp(r, f, t, S): + """Clean fBM vs domain-warped fBM, difference blended.""" + canvas_a = _render_vf(r, "sm", + lambda g, f, t, S: vf_fbm(g, f, t, S, octaves=5, freq=0.04, speed=0.1), + hf_plasma(0.2), PAL_DENSE, f, t, S, sat=0.6) + + canvas_b = _render_vf(r, "md", + lambda g, f, t, S: vf_domain_warp(g, f, t, S, + warp_strength=20.0, freq=0.05, speed=0.15), + hf_time_cycle(0.05), PAL_BRAILLE, f, t, S, sat=0.7) + + return blend_canvas(canvas_a, canvas_b, "difference", 0.7) +``` + +--- + +## Complex — Three Grids + Conditional + Feedback + +### Psychedelic Cathedral + +Three-grid composition with beat-triggered kaleidoscope and feedback zoom tunnel. The most visually complex pattern. + +```python +def fx_cathedral(r, f, t, S): + """Three-layer cathedral: interference + rings + noise, kaleidoscope on beat, + feedback zoom tunnel.""" + # Layer 1: interference pattern on sm grid + canvas_a = _render_vf(r, "sm", + lambda g, f, t, S: vf_interference(g, f, t, S, n_waves=7) * 1.3, + hf_angle(0.0), PAL_MATH, f, t, S, sat=0.8) + + # Layer 2: pulsing rings on md grid + canvas_b = _render_vf(r, "md", + lambda g, f, t, S: vf_rings(g, f, t, S, n_base=10, spacing_base=3) * 1.4, + hf_distance(0.3, 0.02), PAL_STARS, f, t, S, sat=0.9) + + # Layer 3: temporal noise on lg grid (slow morph) + canvas_c = _render_vf(r, "lg", + lambda g, f, t, S: vf_temporal_noise(g, f, t, S, + freq=0.04, t_freq=0.2, octaves=3), + hf_time_cycle(0.12), PAL_BLOCKS, f, t, S, sat=0.7) + + # Blend: A screen B, then difference with C + result = blend_canvas(canvas_a, canvas_b, "screen", 0.8) + result = blend_canvas(result, canvas_c, "difference", 0.5) + + # Beat-triggered kaleidoscope + if f.get("bdecay", 0) > 0.3: + folds = 6 if f.get("sub_r", 0.3) > 0.4 else 8 + result = sh_kaleidoscope(result.copy(), folds=folds) + + return result + +# Scene table entry with feedback: +# {"start": 30.0, "end": 50.0, "name": "cathedral", "fx": fx_cathedral, +# "gamma": 0.65, "shaders": [("bloom", {"thr": 110}), ("chromatic", {"amt": 4}), +# ("vignette", {"s": 0.2}), ("grain", {"amt": 8})], +# "feedback": {"decay": 0.75, "blend": "screen", "opacity": 0.35, +# "transform": "zoom", "transform_amt": 0.012, "hue_shift": 0.015}} +``` + +### Masked Reaction-Diffusion with Attractor Overlay + +Reaction-diffusion visible only through an animated iris mask, with a strange attractor density field underneath. + +```python +def fx_masked_life(r, f, t, S): + """Attractor base + reaction-diffusion visible through iris mask + particles.""" + g_sm = r.get_grid("sm") + g_md = r.get_grid("md") + + # Layer 1: strange attractor density field (background) + canvas_bg = _render_vf(r, "sm", + lambda g, f, t, S: vf_strange_attractor(g, f, t, S, + attractor="clifford", n_points=30000), + hf_time_cycle(0.04), PAL_DOTS, f, t, S, sat=0.5) + + # Layer 2: reaction-diffusion (foreground, will be masked) + canvas_rd = _render_vf(r, "md", + lambda g, f, t, S: vf_reaction_diffusion(g, f, t, S, + feed=0.046, kill=0.063, steps_per_frame=4, init_mode="ring"), + hf_angle(0.15), PAL_HALFFILL, f, t, S, sat=0.85) + + # Animated iris mask — opens over first 5 seconds of scene + scene_start = S.get("_scene_start", t) + if "_scene_start" not in S: + S["_scene_start"] = t + mask = mask_iris(g_md, t, scene_start, scene_start + 5.0, + max_radius=0.6) + canvas_rd = apply_mask_canvas(canvas_rd, mask, bg_canvas=canvas_bg) + + # Layer 3: flow-field particles following the R-D gradient + rd_field = vf_reaction_diffusion(g_sm, f, t, S, + feed=0.046, kill=0.063, steps_per_frame=0) # read without stepping + ch_p, co_p = update_flow_particles(S, g_sm, f, rd_field, + n=300, speed=0.8, char_set=list("·•◦∘°")) + canvas_p = g_sm.render(ch_p, co_p) + + result = blend_canvas(canvas_rd, canvas_p, "add", 0.7) + return result +``` + +### Morphing Field Sequence with Eased Keyframes + +Demonstrates temporal coherence: smooth morphing between effects with keyframed parameters. + +```python +def fx_morphing_journey(r, f, t, S): + """Morphs through 4 value fields over 20 seconds with eased transitions. + Parameters (twist, arm count) also keyframed.""" + # Keyframed twist parameter + twist = keyframe(t, [(0, 1.0), (5, 5.0), (10, 2.0), (15, 8.0), (20, 1.0)], + ease_fn=ease_in_out_cubic, loop=True) + + # Sequence of value fields with 2s crossfade + fields = [ + lambda g, f, t, S: vf_plasma(g, f, t, S), + lambda g, f, t, S: vf_vortex(g, f, t, S, twist=twist), + lambda g, f, t, S: vf_fbm(g, f, t, S, octaves=5, freq=0.04), + lambda g, f, t, S: vf_domain_warp(g, f, t, S, warp_strength=15), + ] + durations = [5.0, 5.0, 5.0, 5.0] + + val_fn = lambda g, f, t, S: vf_sequence(g, f, t, S, fields, durations, + crossfade=2.0) + + # Render with slowly rotating hue + canvas = _render_vf(r, "md", val_fn, hf_time_cycle(0.06), + PAL_DENSE, f, t, S, sat=0.8) + + # Second layer: tiled version of same sequence at smaller grid + tiled_fn = lambda g, f, t, S: vf_sequence( + make_tgrid(g, *uv_tile(g, 3, 3, mirror=True)), + f, t, S, fields, durations, crossfade=2.0) + canvas_b = _render_vf(r, "sm", tiled_fn, hf_angle(0.1), + PAL_RUNE, f, t, S, sat=0.6) + + return blend_canvas(canvas, canvas_b, "screen", 0.5) +``` + +--- + +## Specialized — Unique State Patterns + +### Game of Life with Ghost Trails + +Cellular automaton with analog fade trails. Beat injects random cells. + +```python +def fx_life(r, f, t, S): + """Conway's Game of Life with fading ghost trails. + Beat events inject random live cells for disruption.""" + canvas = _render_vf(r, "sm", + lambda g, f, t, S: vf_game_of_life(g, f, t, S, + rule="life", steps_per_frame=1, fade=0.92, density=0.25), + hf_fixed(0.33), PAL_BLOCKS, f, t, S, sat=0.8) + + # Overlay: coral automaton on lg grid for chunky texture + canvas_b = _render_vf(r, "lg", + lambda g, f, t, S: vf_game_of_life(g, f, t, S, + rule="coral", steps_per_frame=1, fade=0.85, density=0.15, seed=99), + hf_time_cycle(0.1), PAL_HATCH, f, t, S, sat=0.6) + + return blend_canvas(canvas, canvas_b, "screen", 0.5) +``` + +### Boids Flock Over Voronoi + +Emergent swarm movement over a cellular background. + +```python +def fx_boid_swarm(r, f, t, S): + """Flocking boids over animated voronoi cells.""" + # Background: voronoi cells + canvas_bg = _render_vf(r, "md", + lambda g, f, t, S: vf_voronoi(g, f, t, S, + n_cells=20, mode="distance", speed=0.2), + hf_distance(0.4, 0.02), PAL_CIRCUIT, f, t, S, sat=0.5) + + # Foreground: boids + g = r.get_grid("md") + ch_b, co_b = update_boids(S, g, f, n_boids=150, perception=6.0, + max_speed=1.5, char_set=list("▸▹►▻→⟶")) + canvas_boids = g.render(ch_b, co_b) + + # Trails for the boids + # (boid positions are stored in S["boid_x"], S["boid_y"]) + S["px"] = list(S.get("boid_x", [])) + S["py"] = list(S.get("boid_y", [])) + ch_t, co_t = draw_particle_trails(S, g, max_trail=6, fade=0.6) + canvas_trails = g.render(ch_t, co_t) + + result = blend_canvas(canvas_bg, canvas_trails, "add", 0.3) + result = blend_canvas(result, canvas_boids, "add", 0.9) + return result +``` + +### Fire Rising Through SDF Text Stencil + +Fire effect visible only through text letterforms. + +```python +def fx_fire_text(r, f, t, S): + """Fire columns visible through text stencil. Text acts as window.""" + g = r.get_grid("lg") + + # Full-screen fire (will be masked) + canvas_fire = _render_vf(r, "sm", + lambda g, f, t, S: np.clip( + vf_fbm(g, f, t, S, octaves=4, freq=0.08, speed=0.8) * + (1.0 - g.rr / g.rows) * # fade toward top + (0.6 + f.get("bass", 0.3) * 0.8), 0, 1), + hf_fixed(0.05), PAL_BLOCKS, f, t, S, sat=0.9) # fire hue + + # Background: dark domain warp + canvas_bg = _render_vf(r, "md", + lambda g, f, t, S: vf_domain_warp(g, f, t, S, + warp_strength=8, freq=0.03, speed=0.05) * 0.3, + hf_fixed(0.6), PAL_DENSE, f, t, S, sat=0.4) + + # Text stencil mask + mask = mask_text(g, "FIRE", row_frac=0.45) + # Expand vertically for multi-row coverage + for offset in range(-2, 3): + shifted = mask_text(g, "FIRE", row_frac=0.45 + offset / g.rows) + mask = mask_union(mask, shifted) + + canvas_masked = apply_mask_canvas(canvas_fire, mask, bg_canvas=canvas_bg) + return canvas_masked +``` + +### Portrait Mode: Vertical Rain + Quote + +Optimized for 9:16. Uses vertical space for long rain trails and stacked text. + +```python +def fx_portrait_rain_quote(r, f, t, S): + """Portrait-optimized: matrix rain (long vertical trails) with stacked quote. + Designed for 1080x1920 (9:16).""" + g = r.get_grid("md") # ~112x100 in portrait + + # Matrix rain — long trails benefit from portrait's extra rows + ch, co, S = eff_matrix_rain(g, f, t, S, + hue=0.33, bri=0.6, pal=PAL_KATA, speed_base=0.4, speed_beat=2.5) + canvas_rain = g.render(ch, co) + + # Tunnel depth underneath for texture + canvas_tunnel = _render_vf(r, "sm", + lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=3.0, complexity=6) * 0.8, + hf_fixed(0.33), PAL_BLOCKS, f, t, S, sat=0.5) + + result = blend_canvas(canvas_tunnel, canvas_rain, "screen", 0.8) + + # Quote text — portrait layout: short lines, many of them + g_text = r.get_grid("lg") # ~90x80 in portrait + quote_lines = layout_text_portrait( + "The code is the art and the art is the code", + max_chars_per_line=20) + # Center vertically + block_start = (g_text.rows - len(quote_lines)) // 2 + ch_t = np.full((g_text.rows, g_text.cols), " ", dtype="U1") + co_t = np.zeros((g_text.rows, g_text.cols, 3), dtype=np.uint8) + total_chars = sum(len(l) for l in quote_lines) + progress = min(1.0, (t - S.get("_scene_start", t)) / 3.0) + if "_scene_start" not in S: S["_scene_start"] = t + render_typewriter(ch_t, co_t, quote_lines, block_start, g_text.cols, + progress, total_chars, (200, 255, 220), t) + canvas_text = g_text.render(ch_t, co_t) + + result = blend_canvas(result, canvas_text, "add", 0.9) + return result +``` + +--- + +## Scene Table Template + +Wire scenes into a complete video: + +```python +SCENES = [ + {"start": 0.0, "end": 5.0, "name": "coral", + "fx": fx_coral, "grid": "sm", "gamma": 0.70, + "shaders": [("bloom", {"thr": 110}), ("vignette", {"s": 0.2})], + "feedback": {"decay": 0.8, "blend": "screen", "opacity": 0.3, + "transform": "zoom", "transform_amt": 0.01}}, + + {"start": 5.0, "end": 15.0, "name": "tunnel_noise", + "fx": fx_tunnel_noise, "grid": "md", "gamma": 0.75, + "shaders": [("chromatic", {"amt": 3}), ("bloom", {"thr": 120}), + ("scanlines", {"intensity": 0.06}), ("grain", {"amt": 8})], + "feedback": None}, + + {"start": 15.0, "end": 35.0, "name": "cathedral", + "fx": fx_cathedral, "grid": "sm", "gamma": 0.65, + "shaders": [("bloom", {"thr": 100}), ("chromatic", {"amt": 5}), + ("color_wobble", {"amt": 0.2}), ("vignette", {"s": 0.18})], + "feedback": {"decay": 0.75, "blend": "screen", "opacity": 0.35, + "transform": "zoom", "transform_amt": 0.012, "hue_shift": 0.015}}, + + {"start": 35.0, "end": 50.0, "name": "morphing", + "fx": fx_morphing_journey, "grid": "md", "gamma": 0.70, + "shaders": [("bloom", {"thr": 110}), ("grain", {"amt": 6})], + "feedback": {"decay": 0.7, "blend": "screen", "opacity": 0.25, + "transform": "rotate_cw", "transform_amt": 0.003}}, +] +``` diff --git a/skills/creative/ascii-video/references/inputs.md b/skills/creative/ascii-video/references/inputs.md index 2dabc4004..09c929e6b 100644 --- a/skills/creative/ascii-video/references/inputs.md +++ b/skills/creative/ascii-video/references/inputs.md @@ -1,5 +1,14 @@ # Input Sources +**Cross-references:** +- Grid system, resolution presets: `architecture.md` +- Effect building blocks (audio-reactive modulation): `effects.md` +- Scene protocol, SCENES table (feature routing): `scenes.md` +- Shader pipeline, output encoding: `shaders.md` +- Performance tuning (audio chunking, WAV caching): `optimization.md` +- Common bugs (sample rate, dtype, silence handling): `troubleshooting.md` +- Complete scene examples with feature usage: `examples.md` + ## Audio Analysis ### Loading @@ -294,23 +303,73 @@ For narrated videos (testimonials, quotes, storytelling), generate speech audio ### ElevenLabs Voice Generation ```python -import requests +import requests, time, os def generate_tts(text, voice_id, api_key, output_path, model="eleven_multilingual_v2"): - """Generate TTS audio via ElevenLabs API.""" + """Generate TTS audio via ElevenLabs API. Streams response to disk.""" + # Skip if already generated (idempotent re-runs) + if os.path.exists(output_path) and os.path.getsize(output_path) > 1000: + return + url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}" headers = {"xi-api-key": api_key, "Content-Type": "application/json"} - data = {"text": text, "model_id": model, - "voice_settings": {"stability": 0.5, "similarity_boost": 0.75}} - resp = requests.post(url, json=data, headers=headers, timeout=30) + data = { + "text": text, + "model_id": model, + "voice_settings": { + "stability": 0.65, + "similarity_boost": 0.80, + "style": 0.15, + "use_speaker_boost": True, + }, + } + resp = requests.post(url, json=data, headers=headers, stream=True) resp.raise_for_status() with open(output_path, "wb") as f: - f.write(resp.content) + for chunk in resp.iter_content(chunk_size=4096): + f.write(chunk) + time.sleep(0.3) # rate limit: avoid 429s on batch generation +``` + +Voice settings notes: +- `stability` 0.65 gives natural variation without drift. Lower (0.3-0.5) for more expressive reads, higher (0.7-0.9) for monotone/narration. +- `similarity_boost` 0.80 keeps it close to the voice profile. Lower for more generic sound. +- `style` 0.15 adds slight stylistic variation. Keep low (0-0.2) for straightforward reads. +- `use_speaker_boost` True improves clarity at the cost of slightly more processing time. + +### Voice Pool + +ElevenLabs has ~20 built-in voices. Use multiple voices for variety across quotes. Reference pool: + +```python +VOICE_POOL = [ + ("JBFqnCBsd6RMkjVDRZzb", "George"), + ("nPczCjzI2devNBz1zQrb", "Brian"), + ("pqHfZKP75CvOlQylNhV4", "Bill"), + ("CwhRBWXzGAHq8TQ4Fs17", "Roger"), + ("cjVigY5qzO86Huf0OWal", "Eric"), + ("onwK4e9ZLuTAKqWW03F9", "Daniel"), + ("IKne3meq5aSn9XLyUdCD", "Charlie"), + ("iP95p4xoKVk53GoZ742B", "Chris"), + ("bIHbv24MWmeRgasZH58o", "Will"), + ("TX3LPaxmHKxFdv7VOQHJ", "Liam"), + ("SAz9YHcvj6GT2YYXdXww", "River"), + ("EXAVITQu4vr4xnSDxMaL", "Sarah"), + ("Xb7hH8MSUJpSbSDYk0k2", "Alice"), + ("pFZP5JQG7iQjIQuC4Bku", "Lily"), + ("XrExE9yKIg1WjnnlVkGX", "Matilda"), + ("FGY2WhTYpPnrIDTdsKH5", "Laura"), + ("SOYHLrjzK2X1ezoPC6cr", "Harry"), + ("hpp4J3VqNfWAUOO0d1Us", "Bella"), + ("N2lVS1w4EtoT3dr4eOWO", "Callum"), + ("cgSgspJ2msm6clMCkdW9", "Jessica"), + ("pNInz6obpgDQGcFmaJgB", "Adam"), +] ``` ### Voice Assignment -Use multiple voices for variety. Shuffle deterministically so re-runs are consistent: +Shuffle deterministically so re-runs produce the same voice mapping: ```python import random as _rng @@ -318,83 +377,199 @@ import random as _rng def assign_voices(n_quotes, voice_pool, seed=42): """Assign a different voice to each quote, cycling if needed.""" r = _rng.Random(seed) - shuffled = list(voice_pool) - r.shuffle(shuffled) - return [shuffled[i % len(shuffled)] for i in range(n_quotes)] + ids = [v[0] for v in voice_pool] + r.shuffle(ids) + return [ids[i % len(ids)] for i in range(n_quotes)] ``` ### Pronunciation Control -TTS text should be separate from display text. Common fixes: +TTS text must be separate from display text. The display text has line breaks for visual layout; the TTS text is a flat sentence with phonetic fixes. + +Common fixes: - Brand names: spell phonetically ("Nous" -> "Noose", "nginx" -> "engine-x") - Abbreviations: expand ("API" -> "A P I", "CLI" -> "C L I") - Technical terms: add phonetic hints +- Punctuation for pacing: periods create pauses, commas create slight pauses ```python -QUOTES = [("Display text here", "Author")] -QUOTES_TTS = ["TTS text with phonetic spelling here"] +# Display text: line breaks control visual layout +QUOTES = [ + ("It can do far more than the Claws,\nand you don't need to buy a Mac Mini.\nNous Research has a winner here.", "Brian Roemmele"), +] + +# TTS text: flat, phonetically corrected for speech +QUOTES_TTS = [ + "It can do far more than the Claws, and you don't need to buy a Mac Mini. Noose Research has a winner here.", +] # Keep both arrays in sync -- same indices ``` ### Audio Pipeline -1. Generate individual TTS clips (MP3/WAV per quote) -2. Get duration of each clip -3. Calculate timing: speech start/end per quote with gaps +1. Generate individual TTS clips (MP3 per quote, skipping existing) +2. Convert each to WAV (mono, 22050 Hz) for duration measurement and concatenation +3. Calculate timing: intro pad + speech + gaps + outro pad = target duration 4. Concatenate into single TTS track with silence padding 5. Mix with background music ```python -def build_tts_track(tts_clips, target_duration, gap_seconds=2.0): - """Concatenate TTS clips with gaps, pad to target duration.""" - # Get durations +def build_tts_track(tts_clips, target_duration, intro_pad=5.0, outro_pad=4.0): + """Concatenate TTS clips with calculated gaps, pad to target duration. + + Returns: + timing: list of (start_time, end_time, quote_index) tuples + """ + sr = 22050 + + # Convert MP3s to WAV for duration and sample-level concatenation durations = [] for clip in tts_clips: + wav = clip.replace(".mp3", ".wav") + subprocess.run( + ["ffmpeg", "-y", "-i", clip, "-ac", "1", "-ar", str(sr), + "-sample_fmt", "s16", wav], + capture_output=True, check=True) result = subprocess.run( ["ffprobe", "-v", "error", "-show_entries", "format=duration", - "-of", "csv=p=0", clip], + "-of", "csv=p=0", wav], capture_output=True, text=True) durations.append(float(result.stdout.strip())) - - # Calculate timing + + # Calculate gap to fill target duration total_speech = sum(durations) - total_gaps = target_duration - total_speech - gap = max(0.5, total_gaps / (len(tts_clips) + 1)) - - timing = [] # (start, end, quote_index) - t = gap # start after initial gap + n_gaps = len(tts_clips) - 1 + remaining = target_duration - total_speech - intro_pad - outro_pad + gap = max(1.0, remaining / max(1, n_gaps)) + + # Build timing and concatenate samples + timing = [] + t = intro_pad + all_audio = [np.zeros(int(sr * intro_pad), dtype=np.int16)] + for i, dur in enumerate(durations): + wav = tts_clips[i].replace(".mp3", ".wav") + with wave.open(wav) as wf: + samples = np.frombuffer(wf.readframes(wf.getnframes()), dtype=np.int16) timing.append((t, t + dur, i)) - t += dur + gap - - # Concatenate with ffmpeg - # ... silence padding + concat filter + all_audio.append(samples) + t += dur + if i < len(tts_clips) - 1: + all_audio.append(np.zeros(int(sr * gap), dtype=np.int16)) + t += gap + + all_audio.append(np.zeros(int(sr * outro_pad), dtype=np.int16)) + + # Pad or trim to exactly target_duration + full = np.concatenate(all_audio) + target_samples = int(sr * target_duration) + if len(full) < target_samples: + full = np.pad(full, (0, target_samples - len(full))) + else: + full = full[:target_samples] + + # Write concatenated TTS track + with wave.open("tts_full.wav", "w") as wf: + wf.setnchannels(1) + wf.setsampwidth(2) + wf.setframerate(sr) + wf.writeframes(full.tobytes()) + return timing ``` ### Audio Mixing -Mix TTS (center) with background music (wide stereo, low volume): +Mix TTS (center) with background music (wide stereo, low volume). The filter chain: +1. TTS mono duplicated to both channels (centered) +2. BGM loudness-normalized, volume reduced to 15%, stereo widened with `extrastereo` +3. Mixed together with dropout transition for smooth endings ```python def mix_audio(tts_path, bgm_path, output_path, bgm_volume=0.15): """Mix TTS centered with BGM panned wide stereo.""" + filter_complex = ( + # TTS: mono -> stereo center + "[0:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=mono," + "pan=stereo|c0=c0|c1=c0[tts];" + # BGM: normalize loudness, reduce volume, widen stereo + f"[1:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=stereo," + f"loudnorm=I=-16:TP=-1.5:LRA=11," + f"volume={bgm_volume}," + f"extrastereo=m=2.5[bgm];" + # Mix with smooth dropout at end + "[tts][bgm]amix=inputs=2:duration=longest:dropout_transition=3," + "aformat=sample_fmts=s16:sample_rates=44100:channel_layouts=stereo[out]" + ) cmd = [ "ffmpeg", "-y", - "-i", tts_path, # mono TTS - "-i", bgm_path, # stereo BGM - "-filter_complex", - f"[0:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=mono," - f"pan=stereo|c0=c0|c1=c0[tts];" # TTS center - f"[1:a]loudnorm=I=-16:TP=-1.5:LRA=11," - f"volume={bgm_volume}," - f"extrastereo=2.5[bgm];" # BGM wide stereo - f"[tts][bgm]amix=inputs=2:duration=longest[out]", - "-map", "[out]", "-c:a", "pcm_s16le", output_path + "-i", tts_path, + "-i", bgm_path, + "-filter_complex", filter_complex, + "-map", "[out]", output_path, ] subprocess.run(cmd, capture_output=True, check=True) ``` +### Per-Quote Visual Style + +Cycle through visual presets per quote for variety. Each preset defines a background effect, color scheme, and text color: + +```python +QUOTE_STYLES = [ + {"hue": 0.08, "accent": 0.7, "bg": "spiral", "text_rgb": (255, 220, 140)}, # warm gold + {"hue": 0.55, "accent": 0.6, "bg": "rings", "text_rgb": (180, 220, 255)}, # cool blue + {"hue": 0.75, "accent": 0.7, "bg": "wave", "text_rgb": (220, 180, 255)}, # purple + {"hue": 0.35, "accent": 0.6, "bg": "matrix", "text_rgb": (140, 255, 180)}, # green + {"hue": 0.95, "accent": 0.8, "bg": "fire", "text_rgb": (255, 180, 160)}, # red/coral + {"hue": 0.12, "accent": 0.5, "bg": "interference", "text_rgb": (255, 240, 200)}, # amber + {"hue": 0.60, "accent": 0.7, "bg": "tunnel", "text_rgb": (160, 210, 255)}, # cyan + {"hue": 0.45, "accent": 0.6, "bg": "aurora", "text_rgb": (180, 255, 220)}, # teal +] + +style = QUOTE_STYLES[quote_index % len(QUOTE_STYLES)] +``` + +This guarantees no two adjacent quotes share the same look, even without randomness. + +### Typewriter Text Rendering + +Display quote text character-by-character synced to speech progress. Recently revealed characters are brighter, creating a "just typed" glow: + +```python +def render_typewriter(ch, co, lines, block_start, cols, progress, total_chars, text_rgb, t): + """Overlay typewriter text onto character/color grids. + progress: 0.0 (nothing visible) to 1.0 (all text visible).""" + chars_visible = int(total_chars * min(1.0, progress * 1.2)) # slight overshoot for snappy feel + tr, tg, tb = text_rgb + char_count = 0 + for li, line in enumerate(lines): + row = block_start + li + col = (cols - len(line)) // 2 + for ci, c in enumerate(line): + if char_count < chars_visible: + age = chars_visible - char_count + bri_factor = min(1.0, 0.5 + 0.5 / (1 + age * 0.015)) # newer = brighter + hue_shift = math.sin(char_count * 0.3 + t * 2) * 0.05 + stamp(ch, co, c, row, col + ci, + (int(min(255, tr * bri_factor * (1.0 + hue_shift))), + int(min(255, tg * bri_factor)), + int(min(255, tb * bri_factor * (1.0 - hue_shift))))) + char_count += 1 + + # Blinking cursor at insertion point + if progress < 1.0 and int(t * 3) % 2 == 0: + # Find cursor position (char_count == chars_visible) + cc = 0 + for li, line in enumerate(lines): + for ci, c in enumerate(line): + if cc == chars_visible: + stamp(ch, co, "\u258c", block_start + li, + (cols - len(line)) // 2 + ci, (255, 220, 100)) + return + cc += 1 +``` + ### Feature Analysis on Mixed Audio Run the standard audio analysis (FFT, beat detection) on the final mixed track so visual effects react to both TTS and music: @@ -404,4 +579,114 @@ Run the standard audio analysis (FFT, beat detection) on the final mixed track s features = analyze_audio("mixed_final.wav", fps=24) ``` -This means visuals will pulse with both the music beats and the speech energy -- creating natural synchronization. +Visuals pulse with both the music beats and the speech energy. + +--- + +## Audio-Video Sync Verification + +After rendering, verify that visual beat markers align with actual audio beats. Drift accumulates from frame timing errors, ffmpeg concat boundaries, and rounding in `fi / fps`. + +### Beat Timestamp Extraction + +```python +def extract_beat_timestamps(features, fps, threshold=0.5): + """Extract timestamps where beat feature exceeds threshold.""" + beat = features["beat"] + timestamps = [] + for fi in range(len(beat)): + if beat[fi] > threshold: + timestamps.append(fi / fps) + return timestamps + +def extract_visual_beat_timestamps(video_path, fps, brightness_jump=30): + """Detect visual beats by brightness jumps between consecutive frames. + Returns timestamps where mean brightness increases by more than threshold.""" + import subprocess + cmd = ["ffmpeg", "-i", video_path, "-f", "rawvideo", "-pix_fmt", "gray", "-"] + proc = subprocess.run(cmd, capture_output=True) + frames = np.frombuffer(proc.stdout, dtype=np.uint8) + # Infer frame dimensions from total byte count + n_pixels = len(frames) + # For 1080p: 1920*1080 pixels per frame + # Auto-detect from video metadata is more robust: + probe = subprocess.run( + ["ffprobe", "-v", "error", "-select_streams", "v:0", + "-show_entries", "stream=width,height", + "-of", "csv=p=0", video_path], + capture_output=True, text=True) + w, h = map(int, probe.stdout.strip().split(",")) + ppf = w * h # pixels per frame + n_frames = n_pixels // ppf + frames = frames[:n_frames * ppf].reshape(n_frames, ppf) + means = frames.mean(axis=1) + + timestamps = [] + for i in range(1, len(means)): + if means[i] - means[i-1] > brightness_jump: + timestamps.append(i / fps) + return timestamps +``` + +### Sync Report + +```python +def sync_report(audio_beats, visual_beats, tolerance_ms=50): + """Compare audio beat timestamps to visual beat timestamps. + + Args: + audio_beats: list of timestamps (seconds) from audio analysis + visual_beats: list of timestamps (seconds) from video brightness analysis + tolerance_ms: max acceptable drift in milliseconds + + Returns: + dict with matched/unmatched/drift statistics + """ + tolerance = tolerance_ms / 1000.0 + matched = [] + unmatched_audio = [] + unmatched_visual = list(visual_beats) + + for at in audio_beats: + best_match = None + best_delta = float("inf") + for vt in unmatched_visual: + delta = abs(at - vt) + if delta < best_delta: + best_delta = delta + best_match = vt + if best_match is not None and best_delta < tolerance: + matched.append({"audio": at, "visual": best_match, "drift_ms": best_delta * 1000}) + unmatched_visual.remove(best_match) + else: + unmatched_audio.append(at) + + drifts = [m["drift_ms"] for m in matched] + return { + "matched": len(matched), + "unmatched_audio": len(unmatched_audio), + "unmatched_visual": len(unmatched_visual), + "total_audio_beats": len(audio_beats), + "total_visual_beats": len(visual_beats), + "mean_drift_ms": np.mean(drifts) if drifts else 0, + "max_drift_ms": np.max(drifts) if drifts else 0, + "p95_drift_ms": np.percentile(drifts, 95) if len(drifts) > 1 else 0, + } + +# Usage: +audio_beats = extract_beat_timestamps(features, fps=24) +visual_beats = extract_visual_beat_timestamps("output.mp4", fps=24) +report = sync_report(audio_beats, visual_beats) +print(f"Matched: {report['matched']}/{report['total_audio_beats']} beats") +print(f"Mean drift: {report['mean_drift_ms']:.1f}ms, Max: {report['max_drift_ms']:.1f}ms") +# Target: mean drift < 20ms, max drift < 42ms (1 frame at 24fps) +``` + +### Common Sync Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Consistent late visual beats | ffmpeg concat adds frames at boundaries | Use `-vsync cfr` flag; pad segments to exact frame count | +| Drift increases over time | Floating-point accumulation in `t = fi / fps` | Use integer frame counter, compute `t` fresh each frame | +| Random missed beats | Beat threshold too high / feature smoothing too aggressive | Lower threshold; reduce EMA alpha for beat feature | +| Beats land on wrong frame | Off-by-one in frame indexing | Verify: frame 0 = t=0, frame 1 = t=1/fps (not t=0) | diff --git a/skills/creative/ascii-video/references/optimization.md b/skills/creative/ascii-video/references/optimization.md index e7650c227..40ac96fbb 100644 --- a/skills/creative/ascii-video/references/optimization.md +++ b/skills/creative/ascii-video/references/optimization.md @@ -1,5 +1,15 @@ # Optimization Reference +**Cross-references:** +- Grid system, resolution presets, portrait GridLayer: `architecture.md` +- Effect building blocks (pre-computation strategies): `effects.md` +- `_render_vf()`, tonemap (subsampled percentile): `composition.md` +- Scene protocol, render_clip: `scenes.md` +- Shader pipeline, encoding (ffmpeg flags): `shaders.md` +- Input sources (audio chunking, WAV extraction): `inputs.md` +- Common bugs (memory, OOM, frame drops): `troubleshooting.md` +- Complete scene examples: `examples.md` + ## Hardware Detection Detect the user's hardware at script startup and adapt rendering parameters automatically. Never hardcode worker counts or resolution. @@ -124,6 +134,8 @@ def apply_quality_profile(profile): parser = argparse.ArgumentParser() parser.add_argument("--quality", choices=["draft", "preview", "production", "max", "auto"], default="auto", help="Render quality preset") +parser.add_argument("--aspect", choices=["landscape", "portrait", "square"], + default="landscape", help="Aspect ratio preset") parser.add_argument("--workers", type=int, default=0, help="Override worker count (0=auto)") parser.add_argument("--resolution", type=str, default="", help="Override resolution e.g. 1280x720") args = parser.parse_args() @@ -132,6 +144,16 @@ hw = detect_hardware() if args.workers > 0: hw["workers"] = args.workers profile = quality_profile(hw, target_duration, args.quality) + +# Apply aspect ratio preset (before manual resolution override) +ASPECT_PRESETS = { + "landscape": (1920, 1080), + "portrait": (1080, 1920), + "square": (1080, 1080), +} +if args.aspect != "landscape" and not args.resolution: + profile["vw"], profile["vh"] = ASPECT_PRESETS[args.aspect] + if args.resolution: w, h = args.resolution.split("x") profile["vw"], profile["vh"] = int(w), int(h) @@ -142,6 +164,47 @@ log(f"Render: {profile['vw']}x{profile['vh']} @{profile['fps']}fps, " f"CRF {profile['crf']}, {profile['workers']} workers") ``` +### Portrait Mode Considerations + +Portrait (1080x1920) has the same pixel count as landscape 1080p, so performance is equivalent. But composition patterns differ: + +| Concern | Landscape | Portrait | +|---------|-----------|----------| +| Grid cols at `lg` | 160 | 90 | +| Grid rows at `lg` | 45 | 80 | +| Max text line chars | ~50 centered | ~25-30 centered | +| Vertical rain | Short travel | Long, dramatic travel | +| Horizontal spectrum | Full width | Needs rotation or compression | +| Radial effects | Natural circles | Tall ellipses (aspect correction handles this) | +| Particle explosions | Wide spread | Tall spread | +| Text stacking | 3-4 lines comfortable | 8-10 lines comfortable | +| Quote layout | 2-3 wide lines | 5-6 short lines | + +**Portrait-optimized patterns:** +- Vertical rain/matrix effects are naturally enhanced — longer column travel +- Fire columns rise through more screen space +- Rising embers/particles have more vertical runway +- Text can be stacked more aggressively with more lines +- Radial effects work if aspect correction is applied (GridLayer handles this automatically) +- Spectrum bars can be rotated 90 degrees (vertical bars from bottom) + +**Portrait text layout:** +```python +def layout_text_portrait(text, max_chars_per_line=25, grid=None): + """Break text into short lines for portrait display.""" + words = text.split() + lines = []; current = "" + for w in words: + if len(current) + len(w) + 1 > max_chars_per_line: + lines.append(current.strip()) + current = w + " " + else: + current += w + " " + if current.strip(): + lines.append(current.strip()) + return lines +``` + ## Performance Budget Target: 100-200ms per frame (5-10 fps single-threaded, 40-80 fps across 8 workers). @@ -173,6 +236,74 @@ canvas[y:y+ch, x:x+cw] = np.maximum(canvas[y:y+ch, x:x+cw], Collect all characters from all palettes + overlay text into the init set. Lazy-init for any missed characters. +## Pre-Rendered Background Textures + +Alternative to `_render_vf()` for backgrounds where characters don't need to change every frame. Pre-bake a static ASCII texture once at init, then multiply by a per-cell color field each frame. One matrix multiply vs thousands of bitmap blits. + +Use when: background layer uses a fixed character palette and only color/brightness varies per frame. NOT suitable for layers where character selection depends on a changing value field. + +### Init: Bake the Texture + +```python +# In GridLayer.__init__: +self._bg_row_idx = np.clip( + (np.arange(VH) - self.oy) // self.ch, 0, self.rows - 1 +) +self._bg_col_idx = np.clip( + (np.arange(VW) - self.ox) // self.cw, 0, self.cols - 1 +) +self._bg_textures = {} + +def make_bg_texture(self, palette): + """Pre-render a static ASCII texture (grayscale float32) once.""" + if palette not in self._bg_textures: + texture = np.zeros((VH, VW), dtype=np.float32) + rng = random.Random(12345) + ch_list = [c for c in palette if c != " " and c in self.bm] + if not ch_list: + ch_list = list(self.bm.keys())[:5] + for row in range(self.rows): + y = self.oy + row * self.ch + if y + self.ch > VH: + break + for col in range(self.cols): + x = self.ox + col * self.cw + if x + self.cw > VW: + break + bm = self.bm[rng.choice(ch_list)] + texture[y:y+self.ch, x:x+self.cw] = bm + self._bg_textures[palette] = texture + return self._bg_textures[palette] +``` + +### Render: Color Field x Cached Texture + +```python +def render_bg(self, color_field, palette=PAL_CIRCUIT): + """Fast background: pre-rendered ASCII texture * per-cell color field. + color_field: (rows, cols, 3) uint8. Returns (VH, VW, 3) uint8.""" + texture = self.make_bg_texture(palette) + # Expand cell colors to pixel coords via pre-computed index maps + color_px = color_field[ + self._bg_row_idx[:, None], self._bg_col_idx[None, :] + ].astype(np.float32) + return (texture[:, :, None] * color_px).astype(np.uint8) +``` + +### Usage in a Scene + +```python +# Build per-cell color from effect fields (cheap — rows*cols, not VH*VW) +hue = ((t * 0.05 + val * 0.2) % 1.0).astype(np.float32) +R, G, B = hsv2rgb(hue, np.full_like(val, 0.5), val) +color_field = mkc(R, G, B, g.rows, g.cols) # (rows, cols, 3) uint8 + +# Render background — single matrix multiply, no per-cell loop +canvas_bg = g.render_bg(color_field, PAL_DENSE) +``` + +The texture init loop runs once and is cached per palette. Per-frame cost is one fancy-index lookup + one broadcast multiply — orders of magnitude faster than the per-cell bitmap blit loop in `render()` for dense backgrounds. + ## Coordinate Array Caching Pre-compute all grid-relative coordinate arrays at init, not per-frame: @@ -215,8 +346,8 @@ all_rows = [] all_cols = [] all_fades = [] for c in range(cols): - head = int(state["ry"][c]) - trail_len = state["rln"][c] + head = int(S["ry"][c]) + trail_len = S["rln"][c] for i in range(trail_len): row = head - i if 0 <= row < rows: @@ -254,6 +385,57 @@ for fi in range(n_cols): # Now map fire_val to chars and colors in one vectorized pass ``` +## PIL String Rendering for Text-Heavy Scenes + +Alternative to per-cell bitmap blitting when rendering many long text strings (scrolling tickers, typewriter sequences, idea floods). Uses PIL's native `ImageDraw.text()` which renders an entire string in one C call, vs one Python-loop bitmap blit per character. + +Typical win: a scene with 56 ticker rows renders 56 PIL `text()` calls instead of ~10K individual bitmap blits. + +Use when: scene renders many rows of readable text strings. NOT suitable for sparse or spatially-scattered single characters (use normal `render()` for those). + +```python +from PIL import Image, ImageDraw + +def render_text_layer(grid, rows_data, font): + """Render dense text rows via PIL instead of per-cell bitmap blitting. + + Args: + grid: GridLayer instance (for oy, ch, ox, font metrics) + rows_data: list of (row_index, text_string, rgb_tuple) — one per row + font: PIL ImageFont instance (grid.font) + + Returns: + uint8 array (VH, VW, 3) — canvas with rendered text + """ + img = Image.new("RGB", (VW, VH), (0, 0, 0)) + draw = ImageDraw.Draw(img) + for row_idx, text, color in rows_data: + y = grid.oy + row_idx * grid.ch + if y + grid.ch > VH: + break + draw.text((grid.ox, y), text, fill=color, font=font) + return np.array(img) +``` + +### Usage in a Ticker Scene + +```python +# Build ticker data (text + color per row) +rows_data = [] +for row in range(n_tickers): + text = build_ticker_text(row, t) # scrolling substring + color = hsv2rgb_scalar(hue, 0.85, bri) # (R, G, B) tuple + rows_data.append((row, text, color)) + +# One PIL pass instead of thousands of bitmap blits +canvas_tickers = render_text_layer(g_md, rows_data, g_md.font) + +# Blend with other layers normally +result = blend_canvas(canvas_bg, canvas_tickers, "screen", 0.9) +``` + +This is purely a rendering optimization — same visual output, fewer draw calls. The grid's `render()` method is still needed for sparse character fields where characters are placed individually based on value fields. + ## Bloom Optimization **Do NOT use `scipy.ndimage.uniform_filter`** -- measured at 424ms/frame. @@ -433,3 +615,82 @@ Scale with hardware. Baseline: 1080p, 24fps, ~180ms/frame/worker. At 720p: multiply times by ~0.5. At 4K: multiply by ~4. Heavier effects (many particles, dense grids, extra shader passes) add ~20-50%. + +--- + +## Temp File Cleanup + +Rendering generates intermediate files that accumulate across runs. Clean up after the final concat/mux step. + +### Files to Clean + +| File type | Source | Location | +|-----------|--------|----------| +| WAV extracts | `ffmpeg -i input.mp3 ... tmp.wav` | `tempfile.mktemp()` or project dir | +| Segment clips | `render_clip()` output | `segments/seg_00.mp4` etc. | +| Concat list | ffmpeg concat demuxer input | `segments/concat.txt` | +| ffmpeg stderr logs | piped to file for debugging | `*.log` in project dir | +| Feature cache | pickled numpy arrays | `*.pkl` or `*.npz` | + +### Cleanup Function + +```python +import glob +import tempfile +import shutil + +def cleanup_render_artifacts(segments_dir="segments", keep_final=True): + """Remove intermediate files after successful render. + + Call this AFTER verifying the final output exists and plays correctly. + + Args: + segments_dir: directory containing segment clips and concat list + keep_final: if True, only delete intermediates (not the final output) + """ + removed = [] + + # 1. Segment clips + if os.path.isdir(segments_dir): + shutil.rmtree(segments_dir) + removed.append(f"directory: {segments_dir}") + + # 2. Temporary WAV files + for wav in glob.glob("*.wav"): + if wav.startswith("tmp") or wav.startswith("extracted_"): + os.remove(wav) + removed.append(wav) + + # 3. ffmpeg stderr logs + for log in glob.glob("ffmpeg_*.log"): + os.remove(log) + removed.append(log) + + # 4. Feature cache (optional — useful to keep for re-renders) + # for cache in glob.glob("features_*.npz"): + # os.remove(cache) + # removed.append(cache) + + print(f"Cleaned {len(removed)} artifacts: {removed}") + return removed +``` + +### Integration with Render Pipeline + +Call cleanup at the end of the main render script, after the final output is verified: + +```python +# At end of main() +if os.path.exists(output_path) and os.path.getsize(output_path) > 1000: + cleanup_render_artifacts(segments_dir="segments") + print(f"Done. Output: {output_path}") +else: + print("WARNING: final output missing or empty — skipping cleanup") +``` + +### Temp File Best Practices + +- Use `tempfile.mkdtemp()` for segment directories — avoids polluting the project dir +- Name WAV extracts with `tempfile.mktemp(suffix=".wav")` so they're in the OS temp dir +- For debugging, set `KEEP_INTERMEDIATES=1` env var to skip cleanup +- Feature caches (`.npz`) are cheap to store and expensive to recompute — default to keeping them diff --git a/skills/creative/ascii-video/references/scenes.md b/skills/creative/ascii-video/references/scenes.md index 66f48557c..116025327 100644 --- a/skills/creative/ascii-video/references/scenes.md +++ b/skills/creative/ascii-video/references/scenes.md @@ -1,5 +1,15 @@ # Scene System Reference +**Cross-references:** +- Grid system, palettes, color (HSV + OKLAB): `architecture.md` +- Effect building blocks (value fields, noise, SDFs, particles): `effects.md` +- `_render_vf()`, blend modes, tonemap, masking: `composition.md` +- Shader pipeline, feedback buffer, ShaderChain: `shaders.md` +- Complete scene examples at every complexity level: `examples.md` +- Input sources (audio features, video features): `inputs.md` +- Performance tuning, portrait CLI: `optimization.md` +- Common bugs (state leaks, frame drops): `troubleshooting.md` + Scenes are the top-level creative unit. Each scene is a time-bounded segment with its own effect function, shader chain, feedback configuration, and tone-mapping gamma. ## Scene Protocol (v2) @@ -12,7 +22,7 @@ def fx_scene_name(r, f, t, S) -> canvas: Args: r: Renderer instance — access multiple grids via r.get_grid("sm") f: dict of audio/video features, all values normalized to [0, 1] - t: time in seconds (global, not local to scene) + t: time in seconds — local to scene (0.0 at scene start) S: dict for persistent state (particles, rain columns, etc.) Returns: @@ -20,6 +30,20 @@ def fx_scene_name(r, f, t, S) -> canvas: """ ``` +**Local time convention:** Scene functions receive `t` starting at 0.0 for the first frame of the scene, regardless of where the scene appears in the timeline. The render loop subtracts the scene's start time before calling the function: + +```python +# In render_clip: +t_local = fi / FPS - scene_start +canvas = fx_fn(r, feat, t_local, S) +``` + +This makes scenes reorderable without modifying their code. Compute scene progress as: + +```python +progress = min(t / scene_duration, 1.0) # 0→1 over the scene +``` + This replaces the v1 protocol where scenes returned `(chars, colors)` tuples. The v2 protocol gives scenes full control over multi-grid rendering and pixel-level composition internally. ### The Renderer Class diff --git a/skills/creative/ascii-video/references/shaders.md b/skills/creative/ascii-video/references/shaders.md index 83993aa74..84cd5d1df 100644 --- a/skills/creative/ascii-video/references/shaders.md +++ b/skills/creative/ascii-video/references/shaders.md @@ -2,6 +2,15 @@ Post-processing effects applied to the pixel canvas (`numpy uint8 array, shape (H,W,3)`) after character rendering and before encoding. Also covers **pixel-level blend modes**, **feedback buffers**, and the **ShaderChain** compositor. +**Cross-references:** +- Grid system, palettes, color (HSV + OKLAB): `architecture.md` +- Effect building blocks (value fields, noise, SDFs): `effects.md` +- `_render_vf()`, blend modes, tonemap, masking: `composition.md` +- Scene protocol, render_clip, SCENES table: `scenes.md` +- Complete scene examples with shader usage: `examples.md` +- Performance tuning (frame budget, worker count): `optimization.md` +- Encoding pitfalls (ffmpeg flags, color space): `troubleshooting.md` + ## Design Philosophy The shader pipeline turns raw ASCII renders into cinematic output. The system is designed for **composability** — every shader, blend mode, and feedback transform is an independent building block. Combining them creates infinite visual variety from a small set of primitives. @@ -1025,3 +1034,324 @@ cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24", "-vf", f"fps={fps},scale={W}:{H}:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse", "-loop", "0", output_gif] ``` + +### PNG Sequence + +For frame-accurate editing, compositing in external tools (After Effects, Nuke), or lossless archival: + +```python +import os + +def output_png_sequence(frames, output_dir, W, H, fps, prefix="frame"): + """Write frames as numbered PNGs. frames = iterable of uint8 (H,W,3) arrays.""" + os.makedirs(output_dir, exist_ok=True) + + # Method 1: Direct PIL write (no ffmpeg dependency) + from PIL import Image + for i, frame in enumerate(frames): + img = Image.fromarray(frame) + img.save(os.path.join(output_dir, f"{prefix}_{i:06d}.png")) + + # Method 2: ffmpeg pipe (faster for large sequences) + cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24", + "-s", f"{W}x{H}", "-r", str(fps), "-i", "pipe:0", + os.path.join(output_dir, f"{prefix}_%06d.png")] +``` + +Reassemble PNG sequence to video: +```bash +ffmpeg -framerate 24 -i frame_%06d.png -c:v libx264 -crf 18 -pix_fmt yuv420p output.mp4 +``` + +### Alpha Channel / Transparent Background (RGBA) + +For compositing ASCII art over other video or images. Uses RGBA canvas (4 channels) instead of RGB (3 channels): + +```python +def create_rgba_canvas(H, W): + """Transparent canvas — alpha channel starts at 0 (fully transparent).""" + return np.zeros((H, W, 4), dtype=np.uint8) + +def render_char_rgba(canvas, row, col, char_img, color_rgb, alpha=255): + """Render a character with alpha. char_img = PIL glyph mask (grayscale). + Alpha comes from the glyph mask — background stays transparent.""" + r, g, b = color_rgb + y0, x0 = row * cell_h, col * cell_w + mask = np.array(char_img) # grayscale 0-255 + canvas[y0:y0+cell_h, x0:x0+cell_w, 0] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 0], (mask * r / 255).astype(np.uint8)) + canvas[y0:y0+cell_h, x0:x0+cell_w, 1] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 1], (mask * g / 255).astype(np.uint8)) + canvas[y0:y0+cell_h, x0:x0+cell_w, 2] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 2], (mask * b / 255).astype(np.uint8)) + canvas[y0:y0+cell_h, x0:x0+cell_w, 3] = np.maximum(canvas[y0:y0+cell_h, x0:x0+cell_w, 3], mask) + +def blend_onto_background(rgba_canvas, bg_rgb): + """Composite RGBA canvas over a solid or image background.""" + alpha = rgba_canvas[:, :, 3:4].astype(np.float32) / 255.0 + fg = rgba_canvas[:, :, :3].astype(np.float32) + bg = bg_rgb.astype(np.float32) + result = fg * alpha + bg * (1.0 - alpha) + return result.astype(np.uint8) +``` + +RGBA output via ffmpeg (ProRes 4444 for editing, WebM VP9 for web): +```bash +# ProRes 4444 — preserves alpha, widely supported in NLEs +ffmpeg -y -f rawvideo -pix_fmt rgba -s {W}x{H} -r {fps} -i pipe:0 \ + -c:v prores_ks -profile:v 4444 -pix_fmt yuva444p10le output.mov + +# WebM VP9 — alpha support for web/browser compositing +ffmpeg -y -f rawvideo -pix_fmt rgba -s {W}x{H} -r {fps} -i pipe:0 \ + -c:v libvpx-vp9 -pix_fmt yuva420p -crf 30 -b:v 0 output.webm + +# PNG sequence with alpha (lossless) +ffmpeg -y -f rawvideo -pix_fmt rgba -s {W}x{H} -r {fps} -i pipe:0 \ + frame_%06d.png +``` + +**Key constraint**: shaders that operate on `(H,W,3)` arrays need adaptation for RGBA. Either apply shaders to the RGB channels only and preserve alpha, or write RGBA-aware versions: + +```python +def apply_shader_rgba(canvas_rgba, shader_fn, **kwargs): + """Apply an RGB shader to the color channels of an RGBA canvas.""" + rgb = canvas_rgba[:, :, :3] + alpha = canvas_rgba[:, :, 3:4] + rgb_out = shader_fn(rgb, **kwargs) + return np.concatenate([rgb_out, alpha], axis=2) +``` + +--- + +## Real-Time Terminal Rendering + +Live ASCII display in the terminal using ANSI escape codes. Useful for previewing scenes during development, live performances, and interactive parameter tuning. + +### ANSI Color Escape Codes + +```python +def rgb_to_ansi(r, g, b): + """24-bit true color ANSI escape (supported by most modern terminals).""" + return f"\033[38;2;{r};{g};{b}m" + +ANSI_RESET = "\033[0m" +ANSI_CLEAR = "\033[2J\033[H" # clear screen + cursor home +ANSI_HIDE_CURSOR = "\033[?25l" +ANSI_SHOW_CURSOR = "\033[?25h" +``` + +### Frame-to-ANSI Conversion + +```python +def frame_to_ansi(chars, colors): + """Convert char+color arrays to a single ANSI string for terminal output. + + Args: + chars: (rows, cols) array of single characters + colors: (rows, cols, 3) uint8 RGB array + Returns: + str: ANSI-encoded frame ready for sys.stdout.write() + """ + rows, cols = chars.shape + lines = [] + for r in range(rows): + parts = [] + prev_color = None + for c in range(cols): + rgb = tuple(colors[r, c]) + ch = chars[r, c] + if ch == " " or rgb == (0, 0, 0): + parts.append(" ") + else: + if rgb != prev_color: + parts.append(rgb_to_ansi(*rgb)) + prev_color = rgb + parts.append(ch) + parts.append(ANSI_RESET) + lines.append("".join(parts)) + return "\n".join(lines) +``` + +### Optimized: Delta Updates + +Only redraw characters that changed since the last frame. Eliminates redundant terminal writes for static regions: + +```python +def frame_to_ansi_delta(chars, colors, prev_chars, prev_colors): + """Emit ANSI escapes only for cells that changed.""" + rows, cols = chars.shape + parts = [] + for r in range(rows): + for c in range(cols): + if (chars[r, c] != prev_chars[r, c] or + not np.array_equal(colors[r, c], prev_colors[r, c])): + parts.append(f"\033[{r+1};{c+1}H") # move cursor + rgb = tuple(colors[r, c]) + parts.append(rgb_to_ansi(*rgb)) + parts.append(chars[r, c]) + return "".join(parts) +``` + +### Live Render Loop + +```python +import sys +import time + +def render_live(scene_fn, r, fps=24, duration=None): + """Render a scene function live in the terminal. + + Args: + scene_fn: v2 scene function (r, f, t, S) -> canvas + OR v1-style function that populates a grid + r: Renderer instance + fps: target frame rate + duration: seconds to run (None = run until Ctrl+C) + """ + frame_time = 1.0 / fps + S = {} + f = {} # synthesize features or connect to live audio + + sys.stdout.write(ANSI_HIDE_CURSOR + ANSI_CLEAR) + sys.stdout.flush() + + t0 = time.monotonic() + frame_count = 0 + try: + while True: + t = time.monotonic() - t0 + if duration and t > duration: + break + + # Synthesize features from time (or connect to live audio via pyaudio) + f = synthesize_features(t) + + # Render scene — for terminal, use a small grid + g = r.get_grid("sm") + # Option A: v2 scene → extract chars/colors from canvas (reverse render) + # Option B: call effect functions directly for chars/colors + canvas = scene_fn(r, f, t, S) + + # For terminal display, render chars+colors directly + # (bypassing the pixel canvas — terminal uses character cells) + chars, colors = scene_to_terminal(scene_fn, r, f, t, S, g) + + frame_str = ANSI_CLEAR + frame_to_ansi(chars, colors) + sys.stdout.write(frame_str) + sys.stdout.flush() + + # Frame timing + elapsed = time.monotonic() - t0 - (frame_count * frame_time) + sleep_time = frame_time - elapsed + if sleep_time > 0: + time.sleep(sleep_time) + frame_count += 1 + except KeyboardInterrupt: + pass + finally: + sys.stdout.write(ANSI_SHOW_CURSOR + ANSI_RESET + "\n") + sys.stdout.flush() + +def scene_to_terminal(scene_fn, r, f, t, S, g): + """Run effect functions and return (chars, colors) for terminal display. + For terminal mode, skip the pixel canvas and work with character arrays directly.""" + # Effects that return (chars, colors) work directly + # For vf-based effects, render the value field + hue field to chars/colors: + val = vf_plasma(g, f, t, S) + hue = hf_time_cycle(0.08)(g, t) + mask = val > 0.03 + chars = val2char(val, mask, PAL_DENSE) + R, G, B = hsv2rgb(hue, np.full_like(val, 0.8), val) + colors = mkc(R, G, B, g.rows, g.cols) + return chars, colors +``` + +### Curses-Based Rendering (More Robust) + +For full-featured terminal UIs with proper resize handling and input: + +```python +import curses + +def render_curses(scene_fn, r, fps=24): + """Curses-based live renderer with resize handling and key input.""" + + def _main(stdscr): + curses.start_color() + curses.use_default_colors() + curses.curs_set(0) # hide cursor + stdscr.nodelay(True) # non-blocking input + + # Initialize color pairs (curses supports 256 colors) + # Map RGB to nearest curses color pair + color_cache = {} + next_pair = [1] + + def get_color_pair(r, g, b): + key = (r >> 4, g >> 4, b >> 4) # quantize to reduce pairs + if key not in color_cache: + if next_pair[0] < curses.COLOR_PAIRS - 1: + ci = 16 + (r // 51) * 36 + (g // 51) * 6 + (b // 51) # 6x6x6 cube + curses.init_pair(next_pair[0], ci, -1) + color_cache[key] = next_pair[0] + next_pair[0] += 1 + else: + return 0 + return curses.color_pair(color_cache[key]) + + S = {} + f = {} + frame_time = 1.0 / fps + t0 = time.monotonic() + + while True: + t = time.monotonic() - t0 + f = synthesize_features(t) + + # Adapt grid to terminal size + max_y, max_x = stdscr.getmaxyx() + g = r.get_grid_for_size(max_x, max_y) # dynamic grid sizing + + chars, colors = scene_to_terminal(scene_fn, r, f, t, S, g) + rows, cols = chars.shape + + for row in range(min(rows, max_y - 1)): + for col in range(min(cols, max_x - 1)): + ch = chars[row, col] + rgb = tuple(colors[row, col]) + try: + stdscr.addch(row, col, ch, get_color_pair(*rgb)) + except curses.error: + pass # ignore writes outside terminal bounds + + stdscr.refresh() + + # Handle input + key = stdscr.getch() + if key == ord('q'): + break + + time.sleep(max(0, frame_time - (time.monotonic() - t0 - t))) + + curses.wrapper(_main) +``` + +### Terminal Rendering Constraints + +| Constraint | Value | Notes | +|-----------|-------|-------| +| Max practical grid | ~200x60 | Depends on terminal size | +| Color support | 24-bit (modern), 256 (fallback), 16 (minimal) | Check `$COLORTERM` for truecolor | +| Frame rate ceiling | ~30 fps | Terminal I/O is the bottleneck | +| Delta updates | 2-5x faster | Only worth it when <30% of cells change per frame | +| SSH latency | Kills performance | Local terminals only for real-time | + +**Detect color support:** +```python +import os +def get_terminal_color_depth(): + ct = os.environ.get("COLORTERM", "") + if ct in ("truecolor", "24bit"): + return 24 + term = os.environ.get("TERM", "") + if "256color" in term: + return 8 # 256 colors + return 4 # 16 colors basic ANSI +``` diff --git a/skills/creative/ascii-video/references/troubleshooting.md b/skills/creative/ascii-video/references/troubleshooting.md index 6af622c87..07daf320f 100644 --- a/skills/creative/ascii-video/references/troubleshooting.md +++ b/skills/creative/ascii-video/references/troubleshooting.md @@ -1,5 +1,15 @@ # Troubleshooting Reference +**Cross-references:** +- Grid system, palettes, font selection: `architecture.md` +- Effect building blocks (value fields, noise, SDFs): `effects.md` +- `_render_vf()`, blend modes, tonemap: `composition.md` +- Scene protocol, render_clip, SCENES table: `scenes.md` +- Shader pipeline, feedback buffer, encoding: `shaders.md` +- Input sources (audio, video, TTS): `inputs.md` +- Performance tuning, hardware detection: `optimization.md` +- Complete scene examples: `examples.md` + Common bugs, gotchas, and platform-specific issues encountered during ASCII video development. ## NumPy Broadcasting