# Project Notes

*Maintained by Hermes — last updated February 2025*

---

## 1. Kandinsky (Multimodal Transformer)
- **Repo:** https://github.com/samherring99/kandinsky
- **Local path:** `~/Desktop/Projects/kandinsky`
- **Description:** An anything-to-anything transformer combining text, image, and audio modalities. Trains on Pokemon BLIP captions paired with Gen 1 Pokemon audio cries. Uses audio tokenization adapted from nanoGPT.
- **Status:** Early POC. Training code exists (`model.py`) and dataset creation (`create_dataset.py`) works. Audio heads are producing the same sound — unclear if it's a training issue or data issue.
- **TODO:**
  - Debug why audio heads produce identical output
  - Investigate if model needs more training time
  - Design a data pipeline for better/more training data
  - General repo cleanup (requirements.txt, proper CLI, etc.)

---

## 2. NightwingGameSim (LLM → GameBoy ROM Generator)
- **Repo:** https://github.com/samherring99/NightwingGameSim
- **Local path:** `~/Desktop/Projects/NightwingGameSim`
- **Description:** AI-powered pipeline that turns natural language prompts into playable GameBoy ROM files. Generates C code, compiles with GBDK, outputs `.gb` files. Supports Claude API, local Llama, and RAG backends.
- **Status:** Functional — generation pipeline works end-to-end with Claude 4 system prompt. Has tests, docs, examples, and retry logic.
- **TODO:**
  - Harden the repo, clean up structure
  - Build a better testing pipeline
  - Come up with better prompt ideas / examples

---

## 3. ContentBasedMIR (Music Information Retrieval)
- **Repo:** https://github.com/samherring99/ContentBasedMIR
- **Local path:** `~/Desktop/Projects/ContentBasedMIR`
- **Description:** Music similarity analysis using Spotify API track data. Extracts 54 audio features per song and visualizes similarity matrices for music recommendation.
- **Status:** Early stage. Can download Spotify track analysis data and plot similarity matrices. Needs significant expansion.
- **TODO:**
  - Expand analysis pipeline with more features
  - Integrate with text message data for personalized recommendations
  - Build out visualization and exploration tools
  - General modernization (dependencies, structure)

---

## 4. MessageRetrieval (iMessage RAG/SQL)
- **Repo:** https://github.com/samherring99/MessageRetrieval
- **Local path:** `~/Desktop/Projects/MessageRetrieval`
- **Description:** Natural language querying over iMessage data using SQL generation (text2SQL) instead of vector embeddings. Uses LLM-as-Judge pattern for scoring and ranking retrieved messages.
- **Status:** Has initial text2SQL pipeline and summarization tool. Recently worked on with Claude Code. Needs testing.
- **TODO:**
  - Test out the recent Claude Code work
  - Build "iMessage Jarvis" — answer questions about texts
  - Improve SQL generation prompts and accuracy
  - Better error handling and UX

---

## 5. Grailed Embedding Search
- **Repo:** https://github.com/samherring99/grailed-embedding-search
- **Local path:** `~/Desktop/Projects/grailed-embedding-search`
- **Description:** Embedding-based semantic search over Grailed fashion listings. Uses vector similarity to find related items.
- **Status:** Very early — has a basic similarity search implementation. Previously had a more complex version that's being reworked.
- **TODO:**
  - Build out the search pipeline
  - Add scraping/indexing for listings
  - Improve embedding approach
  - Add UI or CLI for exploring results

---

## 6. NightwingNBA (Sports Analytics)
- **Repo:** https://github.com/samherring99/NightwingNBA
- **Local path:** `~/Desktop/Projects/NightwingNBA`
- **Description:** NBA game prediction system. Builds a database of game data, trains a PyTorch model, and makes daily predictions. Has full pipeline: build DB → write data → train → predict.
- **Status:** Functional pipeline exists. Has database building, training, prediction, and daily update scripts.
- **TODO:**
  - Explore and potentially revive
  - Update data sources if stale
  - Improve model accuracy
  - Add visualization/reporting

---

## 7. Stable Audio Sample Explorer
- **Repo:** https://github.com/samherring99/stable-audio-sample-explorer
- **Local path:** `~/Desktop/Projects/stable-audio-sample-explorer`
- **Description:** Tool for exploring audio samples generated by Stable Audio.
- **Status:** 🪦 **Dead** — no active work needed per Sam.

---

## 8. NightwingArt (Art Tools)
- **Repo:** https://github.com/samherring99/NightwingArt
- **Local path:** `~/Desktop/Projects/NightwingArt`
- **Description:** Collection of art tooling scripts — video editing, clip splicing with beat matching, damage effects, and general image manipulation.
- **Status:** Maintenance mode. Tools exist for various effects. Work happens as-needed.
- **TODO:**
  - Add tools as needed for new art projects

---

## 9. Claude-based VST Building ⚠️ *Needs new repo*
- **Description:** Generate VST audio plugins for DAWs from English language prompts. LLM-powered audio plugin creation.
- **Status:** Concept only — no repo exists yet.
- **TODO:**
  - Create repo
  - Research VST SDK / JUCE framework
  - Design prompt → code → compile pipeline

---

## 10. Government Auction Site Scraper ⚠️ *Needs new repo*
- **Description:** Tool that monitors and scrapes government auction sites in San Francisco for deals.
- **Status:** Concept only — no repo exists yet.
- **TODO:**
  - Create repo
  - Research SF government auction sites and their structure
  - Build scraper + notification system

---

## Priority Assessment

| Project | Activity Level | Suggested Priority |
|---------|---------------|-------------------|
| NightwingGameSim | Active | 🔴 High |
| MessageRetrieval | Active | 🔴 High |
| Kandinsky | Active | 🟡 Medium |
| ContentBasedMIR | Exploratory | 🟡 Medium |
| Grailed Embedding Search | Early | 🟡 Medium |
| NightwingNBA | Dormant | 🟢 Low |
| NightwingArt | As-needed | 🟢 Low |
| VST Builder | Concept | 🔵 Future |
| Gov Auction Scraper | Concept | 🔵 Future |
| Stable Audio Explorer | Dead | ⚫ None |