- **Description:** An anything-to-anything transformer combining text, image, and audio modalities. Trains on Pokemon BLIP captions paired with Gen 1 Pokemon audio cries. Uses audio tokenization adapted from nanoGPT.
- **Status:** Early POC. Training code exists (`model.py`) and dataset creation (`create_dataset.py`) works. Audio heads are producing the same sound — unclear if it's a training issue or data issue.
- **TODO:**
- Debug why audio heads produce identical output
- Investigate if model needs more training time
- Design a data pipeline for better/more training data
- General repo cleanup (requirements.txt, proper CLI, etc.)
---
## 2. NightwingGameSim (LLM → GameBoy ROM Generator)
- **Description:** AI-powered pipeline that turns natural language prompts into playable GameBoy ROM files. Generates C code, compiles with GBDK, outputs `.gb` files. Supports Claude API, local Llama, and RAG backends.
- **Status:** Functional — generation pipeline works end-to-end with Claude 4 system prompt. Has tests, docs, examples, and retry logic.
- **TODO:**
- Harden the repo, clean up structure
- Build a better testing pipeline
- Come up with better prompt ideas / examples
---
## 3. ContentBasedMIR (Music Information Retrieval)
- **Description:** Music similarity analysis using Spotify API track data. Extracts 54 audio features per song and visualizes similarity matrices for music recommendation.
- **Status:** Early stage. Can download Spotify track analysis data and plot similarity matrices. Needs significant expansion.
- **TODO:**
- Expand analysis pipeline with more features
- Integrate with text message data for personalized recommendations
- **Description:** Natural language querying over iMessage data using SQL generation (text2SQL) instead of vector embeddings. Uses LLM-as-Judge pattern for scoring and ranking retrieved messages.
- **Status:** Has initial text2SQL pipeline and summarization tool. Recently worked on with Claude Code. Needs testing.
- **TODO:**
- Test out the recent Claude Code work
- Build "iMessage Jarvis" — answer questions about texts
- **Description:** Semantic similarity search over Grailed fashion listings using CLIP embeddings and FAISS. Search by image URL or text description to find visually similar products.
- **Status:** Functional core pipeline. CLIP ViT-B/32 embeds product cover photos into 512-dim vectors, indexed with FAISS cosine similarity. Has CLI, batch embedding, persistent index save/load, and logging.
- **Description:** NBA game prediction system. Builds a database of game data, trains a PyTorch model, and makes daily predictions. Has full pipeline: build DB → write data → train → predict.
- **Status:** Functional pipeline exists. Has database building, training, prediction, and daily update scripts.
- **Description:** Collection of art tooling scripts — video editing, clip splicing with beat matching, damage effects, and general image manipulation.
- **Status:** Maintenance mode. Tools exist for various effects. Work happens as-needed.
- **TODO:**
- Add tools as needed for new art projects
---
## 9. Claude-based VST Building ⚠️ *Needs new repo*
- **Description:** Generate VST audio plugins for DAWs from English language prompts. LLM-powered audio plugin creation.
- **Status:** Concept only — no repo exists yet.
- **TODO:**
- Create repo
- Research VST SDK / JUCE framework
- Design prompt → code → compile pipeline
---
## 10. Government Auction Site Scraper ⚠️ *Needs new repo*
- **Description:** Tool that monitors and scrapes government auction sites in San Francisco for deals.
- **Status:** Concept only — no repo exists yet.
- **TODO:**
- Create repo
- Research SF government auction sites and their structure
