Project Notes

Maintained by Hermes — last updated February 2025

1. Kandinsky (Multimodal Transformer)

Repo: https://github.com/samherring99/kandinsky
Local path: ~/Desktop/Projects/kandinsky
Description: An anything-to-anything transformer combining text, image, and audio modalities. Trains on Pokemon BLIP captions paired with Gen 1 Pokemon audio cries. Uses audio tokenization adapted from nanoGPT.
Status: Early POC. Training code exists (model.py) and dataset creation (create_dataset.py) works. Audio heads are producing the same sound — unclear if it's a training issue or data issue.
TODO:
- Debug why audio heads produce identical output
- Investigate if model needs more training time
- Design a data pipeline for better/more training data
- General repo cleanup (requirements.txt, proper CLI, etc.)

Repo: https://github.com/samherring99/NightwingGameSim
Local path: ~/Desktop/Projects/NightwingGameSim
Description: AI-powered pipeline that turns natural language prompts into playable GameBoy ROM files. Generates C code, compiles with GBDK, outputs .gb files. Supports Claude API, local Llama, and RAG backends.
Status: Functional — generation pipeline works end-to-end with Claude 4 system prompt. Has tests, docs, examples, and retry logic.
TODO:
- Harden the repo, clean up structure
- Build a better testing pipeline
- Come up with better prompt ideas / examples

Repo: https://github.com/samherring99/ContentBasedMIR
Local path: ~/Desktop/Projects/ContentBasedMIR
Description: Music similarity analysis using Spotify API track data. Extracts 54 audio features per song and visualizes similarity matrices for music recommendation.
Status: Early stage. Can download Spotify track analysis data and plot similarity matrices. Needs significant expansion.
TODO:
- Expand analysis pipeline with more features
- Integrate with text message data for personalized recommendations
- Build out visualization and exploration tools
- General modernization (dependencies, structure)

Repo: https://github.com/samherring99/MessageRetrieval
Local path: ~/Desktop/Projects/MessageRetrieval
Description: Natural language querying over iMessage data using SQL generation (text2SQL) instead of vector embeddings. Uses LLM-as-Judge pattern for scoring and ranking retrieved messages.
Status: Has initial text2SQL pipeline and summarization tool. Recently worked on with Claude Code. Needs testing.
TODO:
- Test out the recent Claude Code work
- Build "iMessage Jarvis" — answer questions about texts
- Improve SQL generation prompts and accuracy
- Better error handling and UX

Repo: https://github.com/samherring99/grailed-embedding-search
Local path: ~/Desktop/Projects/grailed-embedding-search
Description: Embedding-based semantic search over Grailed fashion listings. Uses vector similarity to find related items.
Status: Very early — has a basic similarity search implementation. Previously had a more complex version that's being reworked.
TODO:
- Build out the search pipeline
- Add scraping/indexing for listings
- Improve embedding approach
- Add UI or CLI for exploring results

Repo: https://github.com/samherring99/NightwingNBA
Local path: ~/Desktop/Projects/NightwingNBA
Description: NBA game prediction system. Builds a database of game data, trains a PyTorch model, and makes daily predictions. Has full pipeline: build DB → write data → train → predict.
Status: Functional pipeline exists. Has database building, training, prediction, and daily update scripts.
TODO:
- Explore and potentially revive
- Update data sources if stale
- Improve model accuracy
- Add visualization/reporting

Repo: https://github.com/samherring99/NightwingArt
Local path: ~/Desktop/Projects/NightwingArt
Description: Collection of art tooling scripts — video editing, clip splicing with beat matching, damage effects, and general image manipulation.
Status: Maintenance mode. Tools exist for various effects. Work happens as-needed.
TODO:
- Add tools as needed for new art projects

Description: Generate VST audio plugins for DAWs from English language prompts. LLM-powered audio plugin creation.
Status: Concept only — no repo exists yet.
TODO:
- Create repo
- Research VST SDK / JUCE framework
- Design prompt → code → compile pipeline

Description: Tool that monitors and scrapes government auction sites in San Francisco for deals.
Status: Concept only — no repo exists yet.
TODO:
- Create repo
- Research SF government auction sites and their structure
- Build scraper + notification system