Timmy_Foundation/hermes-agent
15
0
Fork 0
Code Issues 247 Pull Requests 14 Actions 1 Packages Projects Releases Wiki Activity

Replace ml-paper-writing with research-paper-writing: full research pipeline skill

Browse Source 
Replaces the writing-focused ml-paper-writing skill (940 lines) with a
complete end-to-end research paper pipeline (1,599 lines SKILL.md + 3,184
lines across 7 reference files).

New content:
- Full 8-phase pipeline: project setup, literature review, experiment
  design, execution/monitoring, analysis, paper drafting, review/revision,
  submission preparation
- Iterative refinement strategy guide from autoreason research (when to use
  autoreason vs critique-and-revise vs single-pass, model selection)
- Hermes agent integration: delegate_task parallel drafting, cronjob
  monitoring, memory/todo state management, skill composition
- Professional LaTeX tooling: microtype, siunitx, TikZ diagram patterns,
  algorithm2e, subcaption, latexdiff, SciencePlots
- Human evaluation design: annotation protocols, inter-annotator agreement,
  crowdsourcing platforms
- Title, Figure 1, conclusion, appendix strategy, page budget management
- Anonymization checklist, rebuttal writing, camera-ready preparation
- AAAI and COLM venue coverage (checklists, reviewer guidelines)

Preserved from ml-paper-writing:
- All writing philosophy (Nanda, Farquhar, Gopen & Swan, Lipton, Perez)
- Citation verification workflow (5-step mandatory process)
- All 6 conference templates (NeurIPS, ICML, ICLR, ACL, AAAI, COLM)
- Conference requirements, format conversion workflow
- Proactivity/collaboration guidance

Bug fixes in inherited reference files:
- BibLaTeX recommendation now correctly says natbib for conferences
- Bare except clauses fixed to except Exception
- Jinja2 template tags removed from citation-workflow.md
- Stale date caveats added to reviewer-guidelines.md
This commit is contained in:
SHL0MS
2026-04-02 16:10:50 -04:00
parent 798a7b99e4
commit b86647c295
54 changed files with 2865 additions and 947 deletions
Download Patch File Download Diff File Expand all files Collapse all files

940
skills/research/ml-paper-writing/SKILL.md
View File

@@ -1,940 +0,0 @@
---
name: ml-paper-writing
description: Write publication-ready ML/AI papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM. Use when drafting papers from research repos, structuring arguments, verifying citations, or preparing camera-ready submissions. Includes LaTeX templates, reviewer guidelines, and citation verification workflows.
version: 1.0.0
author: Orchestra Research
license: MIT
dependencies: [semanticscholar, arxiv, habanero, requests]
metadata:
hermes:
tags: [Academic Writing, NeurIPS, ICML, ICLR, ACL, AAAI, COLM, LaTeX, Paper Writing, Citations, Research]
---
# ML Paper Writing for Top AI Conferences
Expert-level guidance for writing publication-ready papers targeting **NeurIPS, ICML, ICLR, ACL, AAAI, and COLM**. This skill combines writing philosophy from top researchers (Nanda, Farquhar, Karpathy, Lipton, Steinhardt) with practical tools: LaTeX templates, citation verification APIs, and conference checklists.
## Core Philosophy: Collaborative Writing
**Paper writing is collaborative, but Claude should be proactive in delivering drafts.**
The typical workflow starts with a research repository containing code, results, and experimental artifacts. Claude's role is to:
1. **Understand the project** by exploring the repo, results, and existing documentation
2. **Deliver a complete first draft** when confident about the contribution
3. **Search literature** using web search and APIs to find relevant citations
4. **Refine through feedback cycles** when the scientist provides input
5. **Ask for clarification** only when genuinely uncertain about key decisions
**Key Principle**: Be proactive. If the repo and results are clear, deliver a full draft. Don't block waiting for feedback on every section—scientists are busy. Produce something concrete they can react to, then iterate based on their response.
---
## ⚠️ CRITICAL: Never Hallucinate Citations
**This is the most important rule in academic writing with AI assistance.**
### The Problem
AI-generated citations have a **~40% error rate**. Hallucinated references—papers that don't exist, wrong authors, incorrect years, fabricated DOIs—are a serious form of academic misconduct that can result in desk rejection or retraction.
### The Rule
**NEVER generate BibTeX entries from memory. ALWAYS fetch programmatically.**
| Action | ✅ Correct | ❌ Wrong |
|--------|-----------|----------|
| Adding a citation | Search API → verify → fetch BibTeX | Write BibTeX from memory |
| Uncertain about a paper | Mark as `[CITATION NEEDED]` | Guess the reference |
| Can't find exact paper | Note: "placeholder - verify" | Invent similar-sounding paper |
### When You Can't Verify a Citation
If you cannot programmatically verify a citation, you MUST:
```latex
% EXPLICIT PLACEHOLDER - requires human verification
\cite{PLACEHOLDER_author2024_verify_this} % TODO: Verify this citation exists
```
**Always tell the scientist**: "I've marked [X] citations as placeholders that need verification. I could not confirm these papers exist."
### Recommended: Install Exa MCP for Paper Search
For the best paper search experience, install **Exa MCP** which provides real-time academic search:
**Claude Code:**
```bash
claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp"
```
**Cursor / VS Code** (add to MCP settings):
```json
{
"mcpServers": {
"exa": {
"type": "http",
"url": "https://mcp.exa.ai/mcp"
}
}
}
```
Exa MCP enables searches like:
- "Find papers on RLHF for language models published after 2023"
- "Search for transformer architecture papers by Vaswani"
- "Get recent work on sparse autoencoders for interpretability"
Then verify results with Semantic Scholar API and fetch BibTeX via DOI.
---
## Workflow 0: Starting from a Research Repository
When beginning paper writing, start by understanding the project:
```
Project Understanding:
- [ ] Step 1: Explore the repository structure
- [ ] Step 2: Read README, existing docs, and key results
- [ ] Step 3: Identify the main contribution with the scientist
- [ ] Step 4: Find papers already cited in the codebase
- [ ] Step 5: Search for additional relevant literature
- [ ] Step 6: Outline the paper structure together
- [ ] Step 7: Draft sections iteratively with feedback
```
**Step 1: Explore the Repository**
```bash
# Understand project structure
ls -la
find . -name "*.py" | head -20
find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding"
```
Look for:
- `README.md` - Project overview and claims
- `results/`, `outputs/`, `experiments/` - Key findings
- `configs/` - Experimental settings
- Existing `.bib` files or citation references
- Any draft documents or notes
**Step 2: Identify Existing Citations**
Check for papers already referenced in the codebase:
```bash
# Find existing citations
grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py"
find . -name "*.bib"
```
These are high-signal starting points for Related Work—the scientist has already deemed them relevant.
**Step 3: Clarify the Contribution**
Before writing, explicitly confirm with the scientist:
> "Based on my understanding of the repo, the main contribution appears to be [X].
> The key results show [Y]. Is this the framing you want for the paper,
> or should we emphasize different aspects?"
**Never assume the narrative—always verify with the human.**
**Step 4: Search for Additional Literature**
Use web search to find relevant papers:
```
Search queries to try:
- "[main technique] + [application domain]"
- "[baseline method] comparison"
- "[problem name] state-of-the-art"
- Author names from existing citations
```
Then verify and retrieve BibTeX using the citation workflow below.
**Step 5: Deliver a First Draft**
**Be proactive—deliver a complete draft rather than asking permission for each section.**
If the repo provides clear results and the contribution is apparent:
1. Write the full first draft end-to-end
2. Present the complete draft for feedback
3. Iterate based on scientist's response
If genuinely uncertain about framing or major claims:
1. Draft what you can confidently
2. Flag specific uncertainties: "I framed X as the main contribution—let me know if you'd prefer to emphasize Y instead"
3. Continue with the draft rather than blocking
**Questions to include with the draft** (not before):
- "I emphasized X as the main contribution—adjust if needed"
- "I highlighted results A, B, C—let me know if others are more important"
- "Related work section includes [papers]—add any I missed"
---
## When to Use This Skill
Use this skill when:
- **Starting from a research repo** to write a paper
- **Drafting or revising** specific sections
- **Finding and verifying citations** for related work
- **Formatting** for conference submission
- **Resubmitting** to a different venue (format conversion)
- **Iterating** on drafts with scientist feedback
**Always remember**: First drafts are starting points for discussion, not final outputs.
---
## Balancing Proactivity and Collaboration
**Default: Be proactive. Deliver drafts, then iterate.**
| Confidence Level | Action |
|-----------------|--------|
| **High** (clear repo, obvious contribution) | Write full draft, deliver, iterate on feedback |
| **Medium** (some ambiguity) | Write draft with flagged uncertainties, continue |
| **Low** (major unknowns) | Ask 1-2 targeted questions, then draft |
**Draft first, ask with the draft** (not before):
| Section | Draft Autonomously | Flag With Draft |
|---------|-------------------|-----------------|
| Abstract | Yes | "Framed contribution as X—adjust if needed" |
| Introduction | Yes | "Emphasized problem Y—correct if wrong" |
| Methods | Yes | "Included details A, B, C—add missing pieces" |
| Experiments | Yes | "Highlighted results 1, 2, 3—reorder if needed" |
| Related Work | Yes | "Cited papers X, Y, Z—add any I missed" |
**Only block for input when:**
- Target venue is unclear (affects page limits, framing)
- Multiple contradictory framings seem equally valid
- Results seem incomplete or inconsistent
- Explicit request to review before continuing
**Don't block for:**
- Word choice decisions
- Section ordering
- Which specific results to show (make a choice, flag it)
- Citation completeness (draft with what you find, note gaps)
---
## The Narrative Principle
**The single most critical insight**: Your paper is not a collection of experiments—it's a story with one clear contribution supported by evidence.
Every successful ML paper centers on what Neel Nanda calls "the narrative": a short, rigorous, evidence-based technical story with a takeaway readers care about.
**Three Pillars (must be crystal clear by end of introduction):**
| Pillar | Description | Example |
|--------|-------------|---------|
| **The What** | 1-3 specific novel claims within cohesive theme | "We prove that X achieves Y under condition Z" |
| **The Why** | Rigorous empirical evidence supporting claims | Strong baselines, experiments distinguishing hypotheses |
| **The So What** | Why readers should care | Connection to recognized community problems |
**If you cannot state your contribution in one sentence, you don't yet have a paper.**
---
## Paper Structure Workflow
### Workflow 1: Writing a Complete Paper (Iterative)
Copy this checklist and track progress. **Each step involves drafting → feedback → revision:**
```
Paper Writing Progress:
- [ ] Step 1: Define the one-sentence contribution (with scientist)
- [ ] Step 2: Draft Figure 1 → get feedback → revise
- [ ] Step 3: Draft abstract → get feedback → revise
- [ ] Step 4: Draft introduction → get feedback → revise
- [ ] Step 5: Draft methods → get feedback → revise
- [ ] Step 6: Draft experiments → get feedback → revise
- [ ] Step 7: Draft related work → get feedback → revise
- [ ] Step 8: Draft limitations → get feedback → revise
- [ ] Step 9: Complete paper checklist (required)
- [ ] Step 10: Final review cycle and submission
```
**Step 1: Define the One-Sentence Contribution**
**This step requires explicit confirmation from the scientist.**
Before writing anything, articulate and verify:
- What is the single thing your paper contributes?
- What was not obvious or present before your work?
> "I propose framing the contribution as: '[one sentence]'. Does this capture
> what you see as the main takeaway? Should we adjust the emphasis?"
**Step 2: Draft Figure 1**
Figure 1 deserves special attention—many readers skip directly to it.
- Convey core idea, approach, or most compelling result
- Use vector graphics (PDF/EPS for plots)
- Write captions that stand alone without main text
- Ensure readability in black-and-white (8% of men have color vision deficiency)
**Step 3: Write Abstract (5-Sentence Formula)**
From Sebastian Farquhar (DeepMind):
```
1. What you achieved: "We introduce...", "We prove...", "We demonstrate..."
2. Why this is hard and important
3. How you do it (with specialist keywords for discoverability)
4. What evidence you have
5. Your most remarkable number/result
```
**Delete** generic openings like "Large language models have achieved remarkable success..."
**Step 4: Write Introduction (1-1.5 pages max)**
Must include:
- 2-4 bullet contribution list (max 1-2 lines each in two-column format)
- Clear problem statement
- Brief approach overview
- Methods should start by page 2-3 maximum
**Step 5: Methods Section**
Enable reimplementation:
- Conceptual outline or pseudocode
- All hyperparameters listed
- Architectural details sufficient for reproduction
- Present final design decisions; ablations go in experiments
**Step 6: Experiments Section**
For each experiment, explicitly state:
- What claim it supports
- How it connects to main contribution
- Experimental setting (details in appendix)
- What to observe: "the blue line shows X, which demonstrates Y"
Requirements:
- Error bars with methodology (standard deviation vs standard error)
- Hyperparameter search ranges
- Compute infrastructure (GPU type, total hours)
- Seed-setting methods
**Step 7: Related Work**
Organize methodologically, not paper-by-paper:
**Good:** "One line of work uses Floogledoodle's assumption [refs] whereas we use Doobersnoddle's assumption because..."
**Bad:** "Snap et al. introduced X while Crackle et al. introduced Y."
Cite generously—reviewers likely authored relevant papers.
**Step 8: Limitations Section (REQUIRED)**
All major conferences require this. Counter-intuitively, honesty helps:
- Reviewers are instructed not to penalize honest limitation acknowledgment
- Pre-empt criticisms by identifying weaknesses first
- Explain why limitations don't undermine core claims
**Step 9: Paper Checklist**
NeurIPS, ICML, and ICLR all require paper checklists. See [references/checklists.md](references/checklists.md).
---
## Writing Philosophy for Top ML Conferences
**This section distills the most important writing principles from leading ML researchers.** These aren't optional style suggestions—they're what separates accepted papers from rejected ones.
> "A paper is a short, rigorous, evidence-based technical story with a takeaway readers care about." — Neel Nanda
### The Sources Behind This Guidance
This skill synthesizes writing philosophy from researchers who have published extensively at top venues:
| Source | Key Contribution | Link |
|--------|-----------------|------|
| **Neel Nanda** (Google DeepMind) | The Narrative Principle, What/Why/So What framework | [How to Write ML Papers](https://www.alignmentforum.org/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers) |
| **Sebastian Farquhar** (DeepMind) | 5-sentence abstract formula | [How to Write ML Papers](https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/) |
| **Gopen & Swan** | 7 principles of reader expectations | [Science of Scientific Writing](https://cseweb.ucsd.edu/~swanson/papers/science-of-writing.pdf) |
| **Zachary Lipton** | Word choice, eliminating hedging | [Heuristics for Scientific Writing](https://www.approximatelycorrect.com/2018/01/29/heuristics-technical-scientific-writing-machine-learning-perspective/) |
| **Jacob Steinhardt** (UC Berkeley) | Precision, consistent terminology | [Writing Tips](https://bounded-regret.ghost.io/) |
| **Ethan Perez** (Anthropic) | Micro-level clarity tips | [Easy Paper Writing Tips](https://ethanperez.net/easy-paper-writing-tips/) |
| **Andrej Karpathy** | Single contribution focus | Various lectures |
**For deeper dives into any of these, see:**
- [references/writing-guide.md](references/writing-guide.md) - Full explanations with examples
- [references/sources.md](references/sources.md) - Complete bibliography
### Time Allocation (From Neel Nanda)
Spend approximately **equal time** on each of:
1. The abstract
2. The introduction
3. The figures
4. Everything else combined
**Why?** Most reviewers form judgments before reaching your methods. Readers encounter your paper as: **title → abstract → introduction → figures → maybe the rest.**
### Writing Style Guidelines
#### Sentence-Level Clarity (Gopen & Swan's 7 Principles)
These principles are based on how readers actually process prose. Violating them forces readers to spend cognitive effort on structure rather than content.
| Principle | Rule | Example |
|-----------|------|---------|
| **Subject-verb proximity** | Keep subject and verb close | ❌ "The model, which was trained on..., achieves" → ✅ "The model achieves... after training on..." |
| **Stress position** | Place emphasis at sentence ends | ❌ "Accuracy improves by 15% when using attention" → ✅ "When using attention, accuracy improves by **15%**" |
| **Topic position** | Put context first, new info after | ✅ "Given these constraints, we propose..." |
| **Old before new** | Familiar info → unfamiliar info | Link backward, then introduce new |
| **One unit, one function** | Each paragraph makes one point | Split multi-point paragraphs |
| **Action in verb** | Use verbs, not nominalizations | ❌ "We performed an analysis" → ✅ "We analyzed" |
| **Context before new** | Set stage before presenting | Explain before showing equation |
**Full 7 principles with detailed examples:** See [references/writing-guide.md](references/writing-guide.md#the-7-principles-of-reader-expectations)
#### Micro-Level Tips (Ethan Perez)
These small changes accumulate into significantly clearer prose:
- **Minimize pronouns**: ❌ "This shows..." → ✅ "This result shows..."
- **Verbs early**: Position verbs near sentence start
- **Unfold apostrophes**: ❌ "X's Y" → ✅ "The Y of X" (when awkward)
- **Delete filler words**: "actually," "a bit," "very," "really," "basically," "quite," "essentially"
**Full micro-tips with examples:** See [references/writing-guide.md](references/writing-guide.md#micro-level-writing-tips)
#### Word Choice (Zachary Lipton)
- **Be specific**: ❌ "performance" → ✅ "accuracy" or "latency" (say what you mean)
- **Eliminate hedging**: Drop "may" and "can" unless genuinely uncertain
- **Avoid incremental vocabulary**: ❌ "combine," "modify," "expand" → ✅ "develop," "propose," "introduce"
- **Delete intensifiers**: ❌ "provides *very* tight approximation" → ✅ "provides tight approximation"
#### Precision Over Brevity (Jacob Steinhardt)
- **Consistent terminology**: Different terms for same concept creates confusion. Pick one and stick with it.
- **State assumptions formally**: Before theorems, list all assumptions explicitly
- **Intuition + rigor**: Provide intuitive explanations alongside formal proofs
### What Reviewers Actually Read
Understanding reviewer behavior helps prioritize your effort:
| Paper Section | % Reviewers Who Read | Implication |
|---------------|---------------------|-------------|
| Abstract | 100% | Must be perfect |
| Introduction | 90%+ (skimmed) | Front-load contribution |
| Figures | Examined before methods | Figure 1 is critical |
| Methods | Only if interested | Don't bury the lede |
| Appendix | Rarely | Put only supplementary details |
**Bottom line**: If your abstract and intro don't hook reviewers, they may never read your brilliant methods section.
---
## Conference Requirements Quick Reference
| Conference | Page Limit | Extra for Camera-Ready | Key Requirement |
|------------|------------|------------------------|-----------------|
| **NeurIPS 2025** | 9 pages | +0 | Mandatory checklist, lay summary for accepted |
| **ICML 2026** | 8 pages | +1 | Broader Impact Statement required |
| **ICLR 2026** | 9 pages | +1 | LLM disclosure required, reciprocal reviewing |
| **ACL 2025** | 8 pages (long) | varies | Limitations section mandatory |
| **AAAI 2026** | 7 pages | +1 | Strict style file adherence |
| **COLM 2025** | 9 pages | +1 | Focus on language models |
**Universal Requirements:**
- Double-blind review (anonymize submissions)
- References don't count toward page limit
- Appendices unlimited but reviewers not required to read
- LaTeX required for all venues
**LaTeX Templates:** See [templates/](templates/) directory for all conference templates.
---
## Using LaTeX Templates Properly
### Workflow 4: Starting a New Paper from Template
**Always copy the entire template directory first, then write within it.**
```
Template Setup Checklist:
- [ ] Step 1: Copy entire template directory to new project
- [ ] Step 2: Verify template compiles as-is (before any changes)
- [ ] Step 3: Read the template's example content to understand structure
- [ ] Step 4: Replace example content section by section
- [ ] Step 5: Keep template comments/examples as reference until done
- [ ] Step 6: Clean up template artifacts only at the end
```
**Step 1: Copy the Full Template**
```bash
# Create your paper directory with the complete template
cp -r templates/neurips2025/ ~/papers/my-new-paper/
cd ~/papers/my-new-paper/
# Verify structure is complete
ls -la
# Should see: main.tex, neurips.sty, Makefile, etc.
```
**⚠️ IMPORTANT**: Copy the ENTIRE directory, not just `main.tex`. Templates include:
- Style files (`.sty`) - required for compilation
- Bibliography styles (`.bst`) - required for references
- Example content - useful as reference
- Makefiles - for easy compilation
**Step 2: Verify Template Compiles First**
Before making ANY changes, compile the template as-is:
```bash
# Using latexmk (recommended)
latexmk -pdf main.tex
# Or manual compilation
pdflatex main.tex
bibtex main
pdflatex main.tex
pdflatex main.tex
```
If the unmodified template doesn't compile, fix that first. Common issues:
- Missing TeX packages → install via `tlmgr install <package>`
- Wrong TeX distribution → use TeX Live (recommended)
**Step 3: Keep Template Content as Reference**
Don't immediately delete all example content. Instead:
```latex
% KEEP template examples commented out as you write
% This shows you the expected format
% Template example (keep for reference):
% \begin{figure}[t]
% \centering
% \includegraphics[width=0.8\linewidth]{example-image}
% \caption{Template shows caption style}
% \end{figure}
% Your actual figure:
\begin{figure}[t]
\centering
\includegraphics[width=0.8\linewidth]{your-figure.pdf}
\caption{Your caption following the same style.}
\end{figure}
```
**Step 4: Replace Content Section by Section**
Work through the paper systematically:
```
Replacement Order:
1. Title and authors (anonymize for submission)
2. Abstract
3. Introduction
4. Methods
5. Experiments
6. Related Work
7. Conclusion
8. References (your .bib file)
9. Appendix
```
For each section:
1. Read the template's example content
2. Note any special formatting or macros used
3. Replace with your content following the same patterns
4. Compile frequently to catch errors early
**Step 5: Use Template Macros**
Templates often define useful macros. Check the preamble for:
```latex
% Common template macros to use:
\newcommand{\method}{YourMethodName} % Consistent method naming
\newcommand{\eg}{e.g.,\xspace} % Proper abbreviations
\newcommand{\ie}{i.e.,\xspace}
\newcommand{\etal}{\textit{et al.}\xspace}
```
**Step 6: Clean Up Only at the End**
Only remove template artifacts when paper is nearly complete:
```latex
% BEFORE SUBMISSION - remove these:
% - Commented-out template examples
% - Unused packages
% - Template's example figures/tables
% - Lorem ipsum or placeholder text
% KEEP these:
% - All style files (.sty)
% - Bibliography style (.bst)
% - Required packages from template
% - Any custom macros you're using
```
### Template Pitfalls to Avoid
| Pitfall | Problem | Solution |
|---------|---------|----------|
| Copying only `main.tex` | Missing `.sty`, won't compile | Copy entire directory |
| Modifying `.sty` files | Breaks conference formatting | Never edit style files |
| Adding random packages | Conflicts, breaks template | Only add if necessary |
| Deleting template content too early | Lose formatting reference | Keep as comments until done |
| Not compiling frequently | Errors accumulate | Compile after each section |
### Quick Template Reference
| Conference | Main File | Key Style File | Notes |
|------------|-----------|----------------|-------|
| NeurIPS 2025 | `main.tex` | `neurips.sty` | Has Makefile |
| ICML 2026 | `example_paper.tex` | `icml2026.sty` | Includes algorithm packages |
| ICLR 2026 | `iclr2026_conference.tex` | `iclr2026_conference.sty` | Has math_commands.tex |
| ACL | `acl_latex.tex` | `acl.sty` | Strict formatting |
| AAAI 2026 | `aaai2026-unified-template.tex` | `aaai2026.sty` | Very strict compliance |
| COLM 2025 | `colm2025_conference.tex` | `colm2025_conference.sty` | Similar to ICLR |
---
## Conference Resubmission & Format Conversion
When a paper is rejected or withdrawn from one venue and resubmitted to another, format conversion is required. This is a common workflow in ML research.
### Workflow 3: Converting Between Conference Formats
```
Format Conversion Checklist:
- [ ] Step 1: Identify source and target template differences
- [ ] Step 2: Create new project with target template
- [ ] Step 3: Copy content sections (not preamble)
- [ ] Step 4: Adjust page limits and content
- [ ] Step 5: Update conference-specific requirements
- [ ] Step 6: Verify compilation and formatting
```
**Step 1: Key Template Differences**
| From → To | Page Change | Key Adjustments |
|-----------|-------------|-----------------|
| NeurIPS → ICML | 9 → 8 pages | Cut 1 page, add Broader Impact if missing |
| ICML → ICLR | 8 → 9 pages | Can expand experiments, add LLM disclosure |
| NeurIPS → ACL | 9 → 8 pages | Restructure for NLP conventions, add Limitations |
| ICLR → AAAI | 9 → 7 pages | Significant cuts needed, strict style adherence |
| Any → COLM | varies → 9 | Reframe for language model focus |
**Step 2: Content Migration (NOT Template Merge)**
**Never copy LaTeX preambles between templates.** Instead:
```bash
# 1. Start fresh with target template
cp -r templates/icml2026/ new_submission/
# 2. Copy ONLY content sections from old paper
# - Abstract text
# - Section content (between \section{} commands)
# - Figures and tables
# - Bibliography entries
# 3. Paste into target template structure
```
**Step 3: Adjusting for Page Limits**
When cutting pages (e.g., NeurIPS 9 → AAAI 7):
- Move detailed proofs to appendix
- Condense related work (cite surveys instead of individual papers)
- Combine similar experiments into unified tables
- Use smaller figure sizes with subfigures
- Tighten writing: eliminate redundancy, use active voice
When expanding (e.g., ICML 8 → ICLR 9):
- Add ablation studies reviewers requested
- Expand limitations discussion
- Include additional baselines
- Add qualitative examples
**Step 4: Conference-Specific Adjustments**
| Target Venue | Required Additions |
|--------------|-------------------|
| **ICML** | Broader Impact Statement (after conclusion) |
| **ICLR** | LLM usage disclosure, reciprocal reviewing agreement |
| **ACL/EMNLP** | Limitations section (mandatory), Ethics Statement |
| **AAAI** | Strict adherence to style file (no modifications) |
| **NeurIPS** | Paper checklist (appendix), lay summary if accepted |
**Step 5: Update References**
```latex
% Remove self-citations that reveal identity (for blind review)
% Update any "under review" citations to published versions
% Add new relevant work published since last submission
```
**Step 6: Addressing Previous Reviews**
When resubmitting after rejection:
- **Do** address reviewer concerns in the new version
- **Do** add experiments/clarifications reviewers requested
- **Don't** include a "changes from previous submission" section (blind review)
- **Don't** reference the previous submission or reviews
**Common Conversion Pitfalls:**
- ❌ Copying `\usepackage` commands (causes conflicts)
- ❌ Keeping old conference header/footer commands
- ❌ Forgetting to update `\bibliography{}` path
- ❌ Missing conference-specific required sections
- ❌ Exceeding page limit after format change
---
## Citation Workflow (Hallucination Prevention)
**⚠️ CRITICAL**: AI-generated citations have ~40% error rate. **Never write BibTeX from memory.**
### The Golden Rule
```
IF you cannot programmatically fetch a citation:
→ Mark it as [CITATION NEEDED] or [PLACEHOLDER - VERIFY]
→ Tell the scientist explicitly
→ NEVER invent a plausible-sounding reference
```
### Workflow 2: Adding Citations
```
Citation Verification (MANDATORY for every citation):
- [ ] Step 1: Search using Exa MCP or Semantic Scholar API
- [ ] Step 2: Verify paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef)
- [ ] Step 3: Retrieve BibTeX via DOI (programmatically, not from memory)
- [ ] Step 4: Verify the claim you're citing actually appears in the paper
- [ ] Step 5: Add verified BibTeX to bibliography
- [ ] Step 6: If ANY step fails → mark as placeholder, inform scientist
```
**Step 0: Use Exa MCP for Initial Search (Recommended)**
If Exa MCP is installed, use it to find relevant papers:
```
Search: "RLHF language model alignment 2023"
Search: "sparse autoencoders interpretability"
Search: "attention mechanism transformers Vaswani"
```
Then verify each result with Semantic Scholar and fetch BibTeX via DOI.
**Step 1: Search Semantic Scholar**
```python
from semanticscholar import SemanticScholar
sch = SemanticScholar()
results = sch.search_paper("attention mechanism transformers", limit=5)
for paper in results:
print(f"{paper.title} - {paper.paperId}")
print(f" DOI: {paper.externalIds.get('DOI', 'N/A')}")
```
**Step 2: Verify Existence**
Confirm paper appears in at least two sources (Semantic Scholar + CrossRef/arXiv).
**Step 3: Retrieve BibTeX via DOI**
```python
import requests
def doi_to_bibtex(doi: str) -> str:
"""Get verified BibTeX from DOI via CrossRef."""
response = requests.get(
f"https://doi.org/{doi}",
headers={"Accept": "application/x-bibtex"}
)
response.raise_for_status()
return response.text
# Example
bibtex = doi_to_bibtex("10.48550/arXiv.1706.03762")
print(bibtex)
```
**Step 4: Verify Claims**
Before citing for a specific claim, access the paper and confirm the attributed claim actually appears.
**Step 5: Handle Failures Explicitly**
If you cannot verify a citation at ANY step:
```latex
% Option 1: Explicit placeholder
\cite{PLACEHOLDER_smith2023_verify} % TODO: Could not verify - scientist must confirm
% Option 2: Note in text
... as shown in prior work [CITATION NEEDED - could not verify Smith et al. 2023].
```
**Always inform the scientist:**
> "I could not verify the following citations and have marked them as placeholders:
> - Smith et al. 2023 on reward hacking - could not find in Semantic Scholar
> - Jones 2022 on scaling laws - found similar paper but different authors
> Please verify these before submission."
### Summary: Citation Rules
| Situation | Action |
|-----------|--------|
| Found paper, got DOI, fetched BibTeX | ✅ Use the citation |
| Found paper, no DOI | ✅ Use arXiv BibTeX or manual entry from paper |
| Paper exists but can't fetch BibTeX | ⚠️ Mark placeholder, inform scientist |
| Uncertain if paper exists | ❌ Mark `[CITATION NEEDED]`, inform scientist |
| "I think there's a paper about X" | ❌ **NEVER cite** - search first or mark placeholder |
**🚨 NEVER generate BibTeX from memory—always fetch programmatically. 🚨**
See [references/citation-workflow.md](references/citation-workflow.md) for complete API documentation.
---
## Common Issues and Solutions
**Issue: Abstract too generic**
Delete first sentence if it could be prepended to any ML paper. Start with your specific contribution.
**Issue: Introduction exceeds 1.5 pages**
Split background into Related Work. Front-load contribution bullets. Methods should start by page 2-3.
**Issue: Experiments lack explicit claims**
Add sentence before each experiment: "This experiment tests whether [specific claim]..."
**Issue: Reviewers find paper hard to follow**
- Add explicit signposting: "In this section, we show X"
- Use consistent terminology throughout
- Include figure captions that stand alone
**Issue: Missing statistical significance**
Always include:
- Error bars (specify: std dev or std error)
- Number of runs
- Statistical tests if comparing methods
---
## Reviewer Evaluation Criteria
Reviewers assess papers on four dimensions:
| Criterion | What Reviewers Look For |
|-----------|------------------------|
| **Quality** | Technical soundness, well-supported claims |
| **Clarity** | Clear writing, reproducible by experts |
| **Significance** | Community impact, advances understanding |
| **Originality** | New insights (doesn't require new method) |
**Scoring (NeurIPS 6-point scale):**
- 6: Strong Accept - Groundbreaking, flawless
- 5: Accept - Technically solid, high impact
- 4: Borderline Accept - Solid, limited evaluation
- 3: Borderline Reject - Solid but weaknesses outweigh
- 2: Reject - Technical flaws
- 1: Strong Reject - Known results or ethics issues
See [references/reviewer-guidelines.md](references/reviewer-guidelines.md) for detailed reviewer instructions.
---
## Tables and Figures
### Tables
Use `booktabs` LaTeX package for professional tables:
```latex
\usepackage{booktabs}
\begin{tabular}{lcc}
\toprule
Method & Accuracy ↑ & Latency ↓ \\
\midrule
Baseline & 85.2 & 45ms \\
\textbf{Ours} & \textbf{92.1} & 38ms \\
\bottomrule
\end{tabular}
```
**Rules:**
- Bold best value per metric
- Include direction symbols (↑ higher is better, ↓ lower is better)
- Right-align numerical columns
- Consistent decimal precision
### Figures
- **Vector graphics** (PDF, EPS) for all plots and diagrams
- **Raster** (PNG 600 DPI) only for photographs
- Use **colorblind-safe palettes** (Okabe-Ito or Paul Tol)
- Verify **grayscale readability** (8% of men have color vision deficiency)
- **No title inside figure**—the caption serves this function
- **Self-contained captions**—reader should understand without main text
---
## References & Resources
### Reference Documents (Deep Dives)
| Document | Contents |
|----------|----------|
| [writing-guide.md](references/writing-guide.md) | Gopen & Swan 7 principles, Ethan Perez micro-tips, word choice |
| [citation-workflow.md](references/citation-workflow.md) | Citation APIs, Python code, BibTeX management |
| [checklists.md](references/checklists.md) | NeurIPS 16-item, ICML, ICLR, ACL requirements |
| [reviewer-guidelines.md](references/reviewer-guidelines.md) | Evaluation criteria, scoring, rebuttals |
| [sources.md](references/sources.md) | Complete bibliography of all sources |
### LaTeX Templates
Templates in `templates/` directory: **ICML 2026**, **ICLR 2026**, **NeurIPS 2025**, **ACL/EMNLP**, **AAAI 2026**, **COLM 2025**.
**Compiling to PDF:**
- **VS Code/Cursor**: Install LaTeX Workshop extension + TeX Live → Save to auto-compile
- **Command line**: `latexmk -pdf main.tex` or `pdflatex` + `bibtex` workflow
- **Online**: Upload to [Overleaf](https://overleaf.com)
See [templates/README.md](templates/README.md) for detailed setup instructions.
### Key External Sources
**Writing Philosophy:**
- [Neel Nanda: How to Write ML Papers](https://www.alignmentforum.org/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers) - Narrative, "What/Why/So What"
- [Farquhar: How to Write ML Papers](https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/) - 5-sentence abstract
- [Gopen & Swan: Science of Scientific Writing](https://cseweb.ucsd.edu/~swanson/papers/science-of-writing.pdf) - 7 reader expectation principles
- [Lipton: Heuristics for Scientific Writing](https://www.approximatelycorrect.com/2018/01/29/heuristics-technical-scientific-writing-machine-learning-perspective/) - Word choice
- [Perez: Easy Paper Writing Tips](https://ethanperez.net/easy-paper-writing-tips/) - Micro-level clarity
**APIs:** [Semantic Scholar](https://api.semanticscholar.org/api-docs/) | [CrossRef](https://www.crossref.org/documentation/retrieve-metadata/rest-api/) | [arXiv](https://info.arxiv.org/help/api/basics.html)
**Venues:** [NeurIPS](https://neurips.cc/Conferences/2025/PaperInformation/StyleFiles) | [ICML](https://icml.cc/Conferences/2025/AuthorInstructions) | [ICLR](https://iclr.cc/Conferences/2026/AuthorGuide) | [ACL](https://github.com/acl-org/acl-style-files)

1599
skills/research/research-paper-writing/SKILL.md Normal file
View File

File diff suppressed because it is too large Load Diff

394
skills/research/research-paper-writing/references/autoreason-methodology.md Normal file
View File

@@ -0,0 +1,394 @@
# Autoreason: Iterative Refinement Methodology
Complete reference for the autoreason iterative refinement method, derived from experimental results across subjective writing tasks, competitive programming, and four model tiers. Use this when any output (paper draft, experiment script, analysis, task definition) needs iterative improvement.
**Source**: [NousResearch/autoreason](https://github.com/NousResearch/autoreason) — "Autoreason: When Iterative LLM Refinement Works and Why It Fails"
---
## Strategy Selection Guide
### Decision Tree
```
Is the task objectively verifiable (code, math, factual)?
├── YES → Does the model solve it on the first attempt?
│ ├── YES → Use single pass (no refinement needed)
│ └── NO → Use autoreason (structured analysis → reason-informed revision)
│
└── NO (subjective) → What model tier are you using?
├── Weak (Llama 8B, small models)
│ → Single pass. Model too weak for refinement to help.
│ Invest in generation quality, not iteration.
│
├── Mid-tier (Haiku 3.5, Gemini Flash)
│ → Autoreason with stronger judges. This is the sweet spot.
│ Self-refinement DESTROYS weak model outputs — autoreason prevents this.
│
├── Strong (Sonnet 4)
│ → Autoreason for open-ended tasks. Wins 3/5.
│ Critique-and-revise for concrete technical tasks (2/5).
│
└── Frontier (Sonnet 4.6, Opus)
├── Constrained scope? → Autoreason. Wins 2/3 constrained tasks.
└── Unconstrained? → Critique-and-revise or single pass.
Autoreason FAILS on unconstrained frontier tasks (comes last).
```
### Strategy Comparison Table
| Strategy | Best For | Avoid When | Compute (per iteration) |
|----------|----------|------------|------------------------|
| **Single pass** | Frontier models, template tasks, tight budgets | Mid-tier models where quality ceiling is low | 1 call |
| **Critique-and-revise** | Concrete technical requirements (system design, specifications) | Weak models (degrades output), unconstrained subjective tasks | 2 calls |
| **Autoreason** | Mid-tier models, constrained scope, tasks with genuine tradeoffs | Weak models (Llama 8B), frontier + unconstrained | ~6 calls |
| **Best-of-N** | Almost never recommended | Weak models especially — worse than single pass | N calls |
### Why Each Strategy Fails
| Strategy | Failure Mode | Mechanism |
|----------|-------------|-----------|
| **Single pass** | Quality ceiling | No mechanism to improve beyond first attempt |
| **Critique-and-revise** | Progressive degradation | Model hallucinates problems (sycophancy), scope creeps each pass, never declines to change |
| **Best-of-N** | Random selection | Without good ranking signal, more samples = more mediocre options |
| **Autoreason (unconstrained)** | Synthesis drift | Stronger models produce syntheses so consistently preferred that incumbent never stabilizes |
---
## The Autoreason Loop
### Architecture
```
┌──────────────────────────────────────────────────────────┐
│ ITERATION LOOP │
│ │
│ Incumbent A ──► Critic ──► Author B ──► Synthesizer │
│ │ │ │
│ │ ┌───────────────────────┘ │
│ ▼ ▼ │
│ [A] [AB] [B] │
│ │ │ │ │
│ └──────────────┼────────────┘ │
│ ▼ │
│ Judge Panel (blind) │
│ │ │
│ ▼ │
│ Winner │
│ │ │
│ ┌───────┴───────┐ │
│ ▼ ▼ │
│ A wins k=2 B or AB wins │
│ consecutive? → new incumbent │
│ │ │
│ ▼ │
│ CONVERGED │
└──────────────────────────────────────────────────────────┘
```
### Roles
Every role is a **fresh, isolated agent** with no shared context:
| Role | Input | Output | Key Rule |
|------|-------|--------|----------|
| **Critic** | Task + Incumbent A | List of problems | Find problems ONLY. No fixes. No suggestions. |
| **Author B** | Task + A + Critique | Revised version B | Address each criticism. State which problem each change fixes. |
| **Synthesizer** | Task + X + Y (randomized labels) | Synthesis AB | Take strongest elements of each. Not a compromise. |
| **Judge Panel** | Task + A, AB, B (randomized labels + order) | Ranking | Rank best to worst. No authorship stake. |
### Configuration
| Parameter | Value | Rationale |
|-----------|-------|-----------|
| **Convergence k** | 2 | k=1 premature (94% displaced later). k=2 converges 100%, quality plateaus. k=3 fails 24%, 2x cost, no quality gain. |
| **Author temperature** | 0.7-0.8 | Encourages diverse revisions |
| **Judge temperature** | 0.3 | Encourages consistent evaluation |
| **In-loop judges** | 3 | Balance per-pass cost vs evaluation stability |
| **Final evaluation judges** | 7 | Higher statistical power for final comparison |
| **Max tokens** | 4096 | Standard; 8192 for long-form (papers) |
| **Judge type** | Chain-of-thought | 3x faster convergence on some tasks. Always use. |
| **Tiebreak** | Conservative (incumbent wins) | Prevents false positives — A must be genuinely beaten |
| **Max passes** | 25 (constrained), 50 (remedy) | Safety cap; most converge by pass 10-15 |
### Prompts
#### Critic
```
System: You are a critical reviewer. Your only job is to find real problems.
Be specific and concrete. Do not suggest fixes.
User: Find real problems with this proposal. Focus on:
- Things that won't work as described
- Complexity that doesn't pay for itself
- Assumptions that are wrong
- Missing pieces
Do NOT propose fixes. Just the problems.
```
#### Author B
```
System: You are a senior consultant revising a proposal based on specific
criticisms. Address each valid criticism directly. Do not make changes not
motivated by an identified problem.
User: [TASK] + [VERSION A] + [CRITIC OUTPUT]
Revise to address these problems. For each change, state which problem it fixes.
```
#### Synthesizer
```
System: You are given two versions as equal inputs. Take the strongest elements
from each and produce a coherent synthesis. This is not a compromise.
User: [TASK] + [VERSION X] + [VERSION Y]
(labels randomized — synthesizer doesn't know which is incumbent)
```
#### Judge (Chain-of-Thought) — ALWAYS USE THIS VERSION
```
System: You are an independent evaluator. Think carefully before deciding.
User: [TASK] + Three proposals. For each, think step by step:
1. What does it get right?
2. What does it get wrong or miss?
3. Are numbers and claims defensible?
4. Is detail appropriate or bloated?
After reasoning, rank all three.
RANKING: [best], [second], [worst]
```
#### Baseline Prompts (for comparison experiments)
| Baseline | Prompt |
|----------|--------|
| **Conservative** | "Make minimal improvements while preserving what works. Do not add new sections or significantly expand scope." |
| **Improve this** | "Improve this document." (no further guidance) |
| **Harsh critic** | "Critically evaluate and rewrite, fixing all weaknesses you identify." |
| **Critique & revise** | Step 1: "Produce a structured critique. List specific weaknesses." Step 2: "Revise to address each criticism." |
---
## Scoring: Borda Count
Judges rank candidates. Points awarded by rank position:
| Rank | Points (3 candidates) |
|------|----------------------|
| 1st | 3 |
| 2nd | 2 |
| 3rd | 1 |
**Aggregation**: Sum across all judges. Winner = highest total.
**Tiebreak**: Incumbent (A) wins any tie.
**Example** (3 judges):
- Judge 1: AB > A > B → AB gets 3, A gets 2, B gets 1
- Judge 2: A > AB > B → A gets 3, AB gets 2, B gets 1
- Judge 3: AB > B > A → AB gets 3, B gets 2, A gets 1
- Totals: AB=8, A=6, B=4 → AB wins, becomes new incumbent
**Randomization per judge**:
- Candidate labels randomized (A might be called "Proposal X" for one judge, "Proposal Z" for another)
- Presentation order randomized (AB might appear first or last)
- This prevents position bias and label bias
---
## Model Selection Guide
### Empirical Results by Model Tier
| Model | Autoreason Wins | Autoreason Avg Borda | Best Baseline | Margin | Recommendation |
|-------|----------------|---------------------|---------------|--------|----------------|
| **Llama 3.1 8B** | 1/3 | 23.7 | 25.0 (single) | -1.3 | Skip autoreason. Model too weak for diverse candidates. |
| **Gemini 2.0 Flash** | 2/3 | 25.0 | 20.0 (single) | +5.0 | Good candidate. Moderate gains. |
| **Haiku 3.5** | 3/3 | **42.0** | 33.7 (single) | **+8.3** | **Best candidate.** Perfect scores. Baselines actively destroy quality. |
| **Sonnet 4** | 3/5 | 27.8 | 22.4 (C&R) | +5.4 | Good candidate for open tasks. C&R better for technical tasks. |
| **Sonnet 4.6 (unconstrained)** | 0/1 | 7.0 | 31.0 (C&R) | -24.0 | Do NOT use autoreason without constraints. |
| **Sonnet 4.6 (constrained)** | 2/3 | 29.0 | 27.0 (improve) | +2.0 | Use only with scope constraints. |
### The Generation-Evaluation Gap
The core insight: **autoreason's value depends on the gap between a model's generation capability and its self-evaluation capability.**
```
Weak models (Llama 8B):
Generation: Poor | Self-evaluation: Poor
Gap: Small (both bad) → Autoreason can't help, no diverse candidates
Mid-tier models (Haiku, Flash):
Generation: Decent | Self-evaluation: Poor
Gap: LARGE → Autoreason's sweet spot. External eval bridges the gap.
Strong models (Sonnet 4):
Generation: Good | Self-evaluation: Decent
Gap: Moderate → Autoreason helps on 3/5 tasks
Frontier models (Sonnet 4.6):
Generation: Excellent | Self-evaluation: Good
Gap: Small → Simple methods suffice. Autoreason hurts on unconstrained tasks.
```
**Practical rule**: As model costs drop and capabilities improve, today's frontier becomes tomorrow's mid-tier. The generation-evaluation gap is structural, not temporary. Match refinement architecture to the model's position on the capability curve.
### Judge Selection
| Author Model | Recommended Judge | Rationale |
|-------------|------------------|-----------|
| Llama 8B | Don't use autoreason | Model too weak |
| Gemini Flash | Sonnet 4 | Cross-model evaluation works |
| Haiku 3.5 | Sonnet 4 | Strong external eval is the mechanism |
| Haiku 3.5 | Haiku 3.5 (same) | Still works — tournament structure provides value even without strong judges (20.7 vs 18.3 avg Borda) |
| Sonnet 4 | Sonnet 4 (same) | Same-model judges work at this tier |
| Sonnet 4.6 | Sonnet 4.6 (same) | Only with scope constraints |
---
## Scope Constraint Design
### What Makes Autoreason Work on Constrained Tasks
The same model (Sonnet 4.6) goes from **last place** (unconstrained) to **first place** (constrained) with scope constraints. The constraints bound the improvement space so synthesis drift can't accumulate.
### Effective Constraints
| Constraint Type | Example | Why It Works |
|----------------|---------|-------------|
| **Fixed facts** | "Use only these 8 data points, add nothing else" | Bounds information space |
| **Fixed deliverable** | "500-word startup pitch" (not "improve this") | Defines done condition |
| **Fixed structure** | "Exactly 4 sections, each with 3 numbered items" | Prevents structural drift |
| **Fixed change items** | "Address exactly these 3 reviewer concerns" | Bounds modification scope |
### Ineffective Constraints
| Constraint | Why It Fails | What Happens |
|-----------|-------------|-------------|
| Word count alone | Not a scope constraint | False convergence — rejected for length, not quality |
| "Be concise" | Too vague | Ignored after 2-3 passes |
| "Be comprehensive" | Anti-constraint | Invites scope creep |
| No constraints at all | Unbounded improvement space | Synthesis dominates, no convergence |
### Task Categories
| Task Type | Autoreason Works? | Why |
|-----------|-------------------|-----|
| Tasks with genuine tradeoffs (strategy, policy) | Yes | Multiple valid approaches for tournament to select between |
| Constrained writing (pitch, memo, postmortem) | Mostly (2/3) | Bounded scope, clear evaluation criteria |
| Template-filling (incident postmortem) | No | One correct structure, minimal decision space |
| Competitive programming | Yes | Naturally scoped, test suite provides external verification |
| Open-ended unconstrained + frontier model | No | Synthesis drift, no convergence |
---
## Failure Taxonomy
| Failure Mode | Condition | Detection | Evidence |
|-------------|-----------|-----------|----------|
| **Self-correction unreliable** | No external evaluation signal | Baselines degrade below single pass | Haiku baselines: 16.3 avg vs 33.7 single pass |
| **Drift / synthesis dominance** | Unconstrained scope | A wins <15%, AB dominates | Sonnet 4.6 unconstrained: A wins 12%, AB wins 60%+ |
| **Overfitting to visible feedback** | Shallow revision loop (C&R) | High public/private divergence | C&R overfits 32% on hard code problems |
| **No convergence** | Broken judge pipeline | Parsing failures, <3 valid judges | Mixed panel parser failure: 11+ passes |
| **Model too weak** | Insufficient generation diversity | All candidates look similar | Llama 8B wins only 1/3 tasks |
### Recovery Patterns
| Failure | Recovery |
|---------|----------|
| No convergence (drift) | Add scope constraints to the task |
| No convergence (broken judges) | Fix parser, ensure 3 valid judges before continuing |
| Quality degrades with iteration | Switch to single pass or add constraints |
| Model too weak | Use a stronger model for generation, keep weak model for cheap roles |
| Overfitting (code) | Use structured analysis step, not just test feedback |
---
## Code Domain Adaptation
The autoreason method adapts differently for code vs writing:
### Writing Domain
```
Call 1: Critic (find problems in incumbent)
Call 2: Author B (revise based on critique)
Call 3: Synthesizer (merge A and B)
Calls 4-6: Judge Panel (3 blind judges rank A, B, AB)
```
### Code Domain (6-call budget)
```
Call 1: Initial generation
Call 2: Structured analysis (5 points — NO CODE):
- Problem analysis: what does the problem actually require?
- Approach analysis: what approach did we use, is it correct?
- Failure analysis: why did tests fail?
- Alternative approaches: what else could work?
- Edge cases: what inputs might break the solution?
Calls 3-6: Reason-informed revisions
- Each revision must explain WHY it fixes the issue
- Sees test results from public (visible) test cases
```
**Key difference**: The code strategy replaces the judge panel with test-suite evaluation (objective ground truth). The structured analysis step (Call 2) is what drives recovery — it forces reasoning about *why* the approach failed before attempting fixes.
**Results**: Recovery is the mechanism. Among problems where both autoreason and single-pass failed initially, autoreason recovered 62% vs single-pass's 43% (McNemar p=0.041, Cohen's h=0.32).
---
## Applying Autoreason to Paper Writing
The paper itself was refined using autoreason (Section 8 of the paper):
### Setup
- Model: claude-opus-4
- Judges: 3 Opus judges
- Enhancement: Ground-truth critic (access to actual experimental data)
- Result: Converged in 9 passes
### Key Findings for Paper Refinement
1. **Ground-truth critic is essential**: Without ground-truth access, Opus hallucinated a fabricated ablation study, fake confidence intervals, wrong model names, and incorrect role descriptions. With ground-truth access, the critic caught all four on pass 1.
2. **Judge panel integrity matters**: A broken parser in one judge (Gemini output format mismatch) reduced the panel from 3 to 2 judges. This prevented convergence for 11+ passes. Fixing to 3 working judges, the same incumbent converged in 2 passes. A broken judge doesn't add noise — it prevents equilibrium.
### Recommended Setup for Paper Refinement
```
Critic prompt: "You are reviewing a research paper draft. You have access to the
actual experimental results [GROUND TRUTH DATA]. Find factual errors, unsupported
claims, hallucinated results, and structural problems. Do not suggest fixes."
Author B prompt: "Revise this paper draft to fix the identified problems. For each
change, cite the specific problem it addresses. Do not add claims not supported by
the provided experimental data."
Judge prompt (CoT): "Compare three versions of this paper. For each, evaluate:
1. Factual accuracy against the provided results
2. Clarity of the narrative and contribution
3. Whether claims are properly hedged and supported
4. Writing quality (concision, precision, no filler)
After reasoning, rank all three. RANKING: [best], [second], [worst]"
```
### What to Provide as Ground Truth
- All experimental result JSON files
- Statistical test outputs
- Raw numbers for every table and figure
- Configuration files showing exact hyperparameters
- Code that generated the results (for method description accuracy)
---
## Compute Budget Reference
| Method | Calls per Pass | Typical Passes | Total Calls | Relative Cost |
|--------|---------------|----------------|-------------|---------------|
| Single pass | 1 | 1 | 1 | 1x |
| Best-of-N | N | 1 | N | Nx |
| Critique & revise | 2 | 15 | 30 | 30x |
| Autoreason (in-loop) | ~6 | 10-15 | 60-90 | 60-90x |
| Autoreason (with final eval) | ~6 + 7 | 10-15 + 1 | 67-97 | ~80x |
**Cost-quality tradeoff**: Autoreason uses ~6x more compute per pass and typically runs more passes. This is a real tradeoff. The method trades compute for evaluation quality. On constrained tasks with mid-tier models, this tradeoff is strongly positive. On unconstrained tasks with frontier models, it's negative.
**CoT judges reduce cost**: 1 CoT judge provides evaluation quality comparable to 3 standard judges, at ~40% cost savings. Always use CoT judges.

73
skills/research/ml-paper-writing/references/checklists.md → skills/research/research-paper-writing/references/checklists.md
View File

@@ -10,6 +10,8 @@ This reference documents the mandatory checklist requirements for major ML/AI co
- [ICML Paper Checklist](#icml-paper-checklist)
- [ICLR Requirements](#iclr-requirements)
- [ACL Requirements](#acl-requirements)
- [AAAI Requirements](#aaai-requirements)
- [COLM Requirements](#colm-requirements)
- [Universal Pre-Submission Checklist](#universal-pre-submission-checklist)
---
@@ -280,6 +282,77 @@ If applicable:
---
## AAAI Requirements
### Formatting (Strictest of All Venues)
AAAI enforces formatting rules more strictly than any other major venue. Papers that deviate from the template are desk-rejected.
- [ ] Use the **exact** AAAI style file without modification — no `\setlength`, no `\vspace` hacks, no font overrides
- [ ] 7 pages main content (8 for camera-ready with author info)
- [ ] Two-column format, Times font (set by template)
- [ ] References and appendices do not count toward page limit
- [ ] Abstract must be a single paragraph
- [ ] Do not modify margins, column widths, or font sizes
### Required Sections
- [ ] Abstract (single paragraph, no math or citations)
- [ ] Introduction with clear contribution statement
- [ ] References in AAAI format (uses `aaai2026.bst`)
- [ ] Appendix (optional, unlimited)
### Ethics and Reproducibility
- [ ] Broader impact statement (encouraged but not always mandatory — check current year's CFP)
- [ ] Reproducibility details (datasets, code availability)
- [ ] Acknowledge use of AI writing tools if applicable
### Key Differences from Other Venues
- **No separate limitations section required** (unlike ACL), but discussing limitations is recommended
- **Strictest formatting enforcement** — the style checker will reject non-compliant PDFs
- **No paper checklist** like NeurIPS has, but the universal checklist below still applies
- **Unified template** covers main paper and supplementary in the same file
---
## COLM Requirements
### Overview
COLM (Conference on Language Modeling) focuses specifically on language model research. Framing must target this community.
### Formatting
- [ ] 9 pages main content (10 for camera-ready)
- [ ] Use COLM template (based on ICLR template with modifications)
- [ ] Double-blind review
- [ ] References and appendices unlimited
### Required Sections
- [ ] Abstract
- [ ] Introduction framed for language modeling community
- [ ] Conclusion
- [ ] References
### Content Expectations
- [ ] Contribution must be relevant to language models (broadly interpreted: training, evaluation, applications, theory, alignment, safety)
- [ ] If the method is general, frame with language model examples
- [ ] Baselines should include recent LM-specific methods where applicable
### Key Differences from Other Venues
- **Narrower scope** than NeurIPS/ICML — must frame for LM community
- **Template derived from ICLR** — similar formatting rules
- **Newer venue** — reviewer norms are still establishing; err on the side of thorough evaluation
- **No mandatory checklist** like NeurIPS, but broader impact discussion is expected
- **LLM disclosure**: If LLMs were used in research (code generation, data annotation, writing assistance), disclose this
---
## Universal Pre-Submission Checklist
### Before Every Submission

8
skills/research/ml-paper-writing/references/citation-workflow.md → skills/research/research-paper-writing/references/citation-workflow.md
View File

@@ -289,7 +289,7 @@ class CitationManager:
)
if resp.status_code == 200:
sources.append("CrossRef")
except:
except Exception:
pass
# Check arXiv if ID available
@@ -301,7 +301,7 @@ class CitationManager:
)
if "<entry>" in resp.text and "<title>" in resp.text:
sources.append("arXiv")
except:
except Exception:
pass
return len(sources) >= 2, sources
@@ -318,7 +318,7 @@ class CitationManager:
)
if resp.status_code == 200:
return resp.text
except:
except Exception:
pass
# Fallback: generate from paper data
@@ -419,7 +419,7 @@ def batch_cite(queries: List[str], output_file: str = "references.bib"):
| Customization | Limited | Highly flexible |
| Backend | bibtex | Biber (recommended) |
**Recommendation**: Use BibLaTeX with Biber for new papers.
**Recommendation**: Use natbib with BibTeX for conference submissions — all major venue templates (NeurIPS, ICML, ICLR, ACL, AAAI, COLM) ship with natbib and `.bst` files. BibLaTeX with Biber is an option for journals or personal projects where you control the template.
### LaTeX Setup

728
skills/research/research-paper-writing/references/experiment-patterns.md Normal file
View File

@@ -0,0 +1,728 @@
# Experiment Design Patterns
Patterns and best practices distilled from running research experiments at scale with the Hermes agent. These cover experiment infrastructure, evaluation protocols, monitoring, and failure recovery.
---
## Experiment Infrastructure
### Directory Structure
Organize experiments with a consistent structure:
```
workspace/
experiments/
run_main.py # Core experiment runner
run_baselines.py # Baseline comparison
run_ablation.py # Ablation studies
strategies.py # Method implementations
config.yaml # Shared configuration
results/
<experiment_name>/
<task_or_problem>/
<strategy>/
result.json # Final metrics
final_output.md # Final output artifact
history.json # Full trajectory/log
pass_01/ # Per-iteration artifacts (if iterative)
intermediate.md
analysis/
analyze_results.py # Statistical analysis
compute_stats.py # Significance tests
make_charts.py # Visualization
paper/
paper.tex # LaTeX source
fig_*.pdf # Generated figures
```
### Script Design Principles
**1. Incremental Saving (Crash Recovery)**
Every experiment script should save results after each unit of work, and skip already-completed work on restart:
```python
import json, os
from pathlib import Path
def run_experiment(problems, strategies, output_dir):
for problem in problems:
for strategy in strategies:
result_path = Path(output_dir) / problem["id"] / strategy / "result.json"
if result_path.exists():
print(f"Skipping {problem['id']}/{strategy} (already done)")
continue
# Run the experiment
result = execute_strategy(problem, strategy)
# Save immediately
result_path.parent.mkdir(parents=True, exist_ok=True)
with open(result_path, 'w') as f:
json.dump(result, f, indent=2)
```
This pattern makes re-runs safe and efficient. If a process crashes at problem 47/150, restarting skips the first 46.
**2. Artifact Preservation**
Save all intermediate outputs, not just final results. This enables post-hoc analysis without re-running:
```python
def save_pass_artifacts(output_dir, pass_num, artifacts):
"""Save all artifacts from a single pass of an iterative method."""
pass_dir = Path(output_dir) / f"pass_{pass_num:02d}"
pass_dir.mkdir(parents=True, exist_ok=True)
for name, content in artifacts.items():
with open(pass_dir / f"{name}.md", 'w') as f:
f.write(content)
```
**3. Configuration Management**
Use YAML configs for reproducibility:
```yaml
# config.yaml
model: anthropic/claude-sonnet-4-20250514
author_temperature: 0.8
judge_temperature: 0.3
max_tokens: 4096
num_judges: 3
max_passes: 15
convergence_k: 2
```
```python
import yaml
with open("config.yaml") as f:
config = yaml.safe_load(f)
```
**4. Separation of Concerns**
Keep generation, evaluation, and visualization in separate scripts:
| Script | Purpose |
|--------|---------|
| `run_experiment.py` | Core method execution |
| `run_baselines.py` | Baseline comparisons at same compute |
| `run_eval.py` | Blind evaluation / judge panels |
| `analyze_results.py` | Statistical analysis |
| `make_charts.py` | Figure generation |
This lets you re-run evaluation without re-running expensive generation, and regenerate figures without re-running analysis.
---
## Evaluation Protocols
### Blind Judge Panels (for Subjective Tasks)
When evaluating subjective outputs (writing, analysis, recommendations), use a blind judge panel:
```python
import random
def run_blind_evaluation(outputs: dict, task_prompt: str, num_judges: int = 7):
"""
Run blind evaluation of multiple method outputs.
Args:
outputs: {"method_name": "output_text", ...}
task_prompt: The original task description
num_judges: Number of independent judge evaluations
"""
rankings = []
for judge_i in range(num_judges):
# Randomize labels and presentation order per judge
methods = list(outputs.keys())
random.shuffle(methods)
labels = {m: chr(65 + i) for i, m in enumerate(methods)} # A, B, C...
# Present to judge with randomized labels
prompt = f"Task: {task_prompt}\n\n"
for method in methods:
prompt += f"--- Proposal {labels[method]} ---\n{outputs[method]}\n\n"
prompt += "Rank all proposals from best to worst. Format: RANKING: [best], [second], [worst]"
ranking = call_judge(prompt)
rankings.append({"labels": labels, "ranking": ranking})
# Aggregate via Borda count
return compute_borda(rankings)
def compute_borda(rankings, n_methods=3):
"""Borda count: 3/2/1 points for 1st/2nd/3rd."""
scores = {}
points = {0: n_methods, 1: n_methods - 1, 2: n_methods - 2} # Adjust for n_methods
for r in rankings:
for position, method in enumerate(r["ranking"]):
scores[method] = scores.get(method, 0) + points.get(position, 0)
return scores
```
Key design decisions:
- **Randomize both labels AND order** per judge to prevent position bias
- **Use odd number of judges** (3, 5, 7) to break ties
- **Conservative tiebreak**: Incumbent/baseline wins ties (prevents false positives)
- **CoT judges** match non-CoT quality at ~40% cost (1 CoT judge ≈ 3 standard judges)
### Code/Objective Evaluation
For tasks with ground-truth evaluation (code, math, factual):
```python
import subprocess
def evaluate_code(solution: str, test_cases: list, timeout: int = 30):
"""Run code solution against test cases with sandboxed execution."""
results = {"public": [], "private": []}
for test in test_cases:
try:
proc = subprocess.run(
["python3", "-c", solution],
input=test["input"],
capture_output=True,
timeout=timeout,
text=True
)
actual = proc.stdout.strip()
expected = test["expected"].strip()
passed = actual == expected
except subprocess.TimeoutExpired:
passed = False
category = "public" if test.get("public") else "private"
results[category].append(passed)
return {
"public_pass_rate": sum(results["public"]) / max(len(results["public"]), 1),
"private_pass_rate": sum(results["private"]) / max(len(results["private"]), 1),
}
```
### Compute-Matched Comparison
Always compare methods at equal compute budget. If your method uses N API calls, baselines get N calls too:
| Method | Call Budget | Allocation |
|--------|-----------|------------|
| Single pass | 6 calls | 6 independent generations |
| Critique & revise | 6 calls | 1 generate + 5 revise rounds |
| Autoreason | 6 calls | 1 generate + 1 analysis + 4 revisions |
| Best-of-N | 6 calls | 6 independent, pick best on public test |
### Human Evaluation Design
Many ML/NLP papers require human evaluation, especially for subjective tasks (text generation, summarization, dialogue, creative writing). Poorly designed human evals are a common rejection reason.
#### When Human Evaluation Is Required
| Task Type | Required? | Notes |
|-----------|-----------|-------|
| Text generation (open-ended) | Yes | LLM-as-judge alone is insufficient for acceptance at ACL/EMNLP |
| Summarization | Usually | At minimum for a subset of outputs |
| Dialogue systems | Yes | User studies or annotation |
| Code generation | No | Test suites are objective ground truth |
| Classification | No | Standard metrics suffice |
| Any task with subjective quality | Strongly recommended | Strengthens the paper significantly |
#### Annotation Protocol Design
```
Human Evaluation Protocol:
1. Define the evaluation dimensions (fluency, relevance, factual accuracy, etc.)
2. Create annotation guidelines with examples of each score level
3. Run a pilot with 2-3 annotators on 20-30 examples
4. Compute pilot inter-annotator agreement — if low, revise guidelines
5. Run full evaluation
6. Report: annotator count, agreement metrics, compensation, time per item
```
**Evaluation dimensions** (pick relevant subset):
| Dimension | Definition | Scale |
|-----------|-----------|-------|
| Fluency | Grammaticality and naturalness | 1-5 Likert |
| Relevance | Does it address the task? | 1-5 Likert |
| Factual accuracy | Are stated facts correct? | Binary or 1-5 |
| Coherence | Logical flow and consistency | 1-5 Likert |
| Informativeness | Does it provide useful information? | 1-5 Likert |
| Overall preference | Which output is better? | A/B/Tie (pairwise) |
**Pairwise comparison** (preferred over absolute scoring — more reliable):
- Present two outputs side-by-side (randomize left/right position)
- Ask: "Which is better? A / B / Tie"
- More discriminative and less susceptible to annotator calibration drift
#### Inter-Annotator Agreement
Always report agreement metrics. Without them, reviewers assume your annotations are unreliable.
```python
# Krippendorff's alpha (preferred — handles missing data, any scale)
# pip install krippendorffs-alpha
import krippendorff
# Ratings: rows = annotators, columns = items, values = scores
ratings = [
[3, 4, 1, 2, 5, None, 3], # Annotator 1
[3, 5, 1, 3, 5, 2, 3], # Annotator 2
[4, 4, 2, 2, 4, 2, None], # Annotator 3
]
alpha = krippendorff.alpha(reliability_data=ratings, level_of_measurement="ordinal")
print(f"Krippendorff's alpha: {alpha:.3f}")
# Interpretation: >0.80 good, 0.67-0.80 acceptable, <0.67 questionable
```
```python
# Cohen's kappa (for exactly 2 annotators, categorical data)
from sklearn.metrics import cohen_kappa_score
annotator_1 = [1, 2, 3, 1, 2, 3, 2]
annotator_2 = [1, 2, 2, 1, 3, 3, 2]
kappa = cohen_kappa_score(annotator_1, annotator_2)
print(f"Cohen's kappa: {kappa:.3f}")
# Interpretation: >0.80 excellent, 0.60-0.80 substantial, 0.40-0.60 moderate
```
| Metric | When to Use | Annotators | Scale |
|--------|------------|-----------|-------|
| Krippendorff's alpha | Default choice | Any number | Any (ordinal, nominal, ratio) |
| Cohen's kappa | 2 annotators, categorical | Exactly 2 | Nominal/ordinal |
| Fleiss' kappa | 3+ annotators, categorical | 3+ | Nominal |
| Pearson/Spearman | Continuous scores | 2 | Interval/ratio |
#### Crowdsourcing Platforms
| Platform | Best For | Cost | Quality |
|----------|----------|------|---------|
| **Prolific** | Academic research, higher quality | $8-15/hr | High — academic participant pool |
| **MTurk** | Large-scale, fast turnaround | $2-10/hr | Variable — use qualifications |
| **Surge AI** | NLP-specific annotations | Premium | High — trained annotators |
| **Expert annotators** | Domain-specific (medical, legal) | Highest | Highest — but slow |
**Ethics requirements**:
- Report compensation rate (must be at minimum local minimum wage)
- Describe annotator demographics if relevant
- Obtain IRB/ethics approval if required by your institution
- ACL venues explicitly require compensation documentation
#### What to Report in the Paper
```
Human Evaluation Section Checklist:
- [ ] Number of annotators
- [ ] Annotator qualifications / recruitment method
- [ ] Number of items evaluated
- [ ] Evaluation dimensions with definitions
- [ ] Scale used (Likert, pairwise, binary)
- [ ] Inter-annotator agreement (Krippendorff's alpha or Cohen's kappa)
- [ ] Compensation rate
- [ ] Time per annotation item
- [ ] Whether annotators saw model identities (should be blind)
- [ ] Randomization of presentation order
```
---
## Statistical Analysis
### Required Tests
| Test | When to Use | Python |
|------|------------|--------|
| McNemar's test | Comparing two methods on same problems | `scipy.stats.binomtest` for small n |
| Two-proportion z-test | Comparing success rates | Custom or `statsmodels` |
| Fisher's exact test | Small sample pairwise comparison | `scipy.stats.fisher_exact` |
| Bootstrapped CI | Confidence intervals for any metric | Custom bootstrap |
| Cohen's h | Effect size for proportions | Manual calculation |
### Standard Analysis Script
```python
import numpy as np
from scipy import stats
from pathlib import Path
import json
def load_all_results(results_dir):
"""Load all results into a structured format."""
results = {}
for result_file in Path(results_dir).rglob("result.json"):
parts = result_file.relative_to(results_dir).parts
if len(parts) >= 3:
experiment, task, strategy = parts[0], parts[1], parts[2]
data = json.loads(result_file.read_text())
results.setdefault(experiment, {}).setdefault(strategy, {})[task] = data
return results
def pairwise_mcnemar(method_a_results, method_b_results):
"""McNemar's test for paired binary outcomes."""
a_win_b_lose = sum(1 for a, b in zip(method_a_results, method_b_results) if a and not b)
b_win_a_lose = sum(1 for a, b in zip(method_a_results, method_b_results) if b and not a)
n = a_win_b_lose + b_win_a_lose
if n < 25:
# Use exact binomial for small samples
result = stats.binomtest(a_win_b_lose, n, 0.5)
p_value = result.pvalue
else:
# Chi-squared approximation
chi2 = (abs(a_win_b_lose - b_win_a_lose) - 1)**2 / (a_win_b_lose + b_win_a_lose)
p_value = 1 - stats.chi2.cdf(chi2, df=1)
return {
"a_wins": a_win_b_lose,
"b_wins": b_win_a_lose,
"n_discordant": n,
"p_value": p_value,
"significant": p_value < 0.05
}
def bootstrap_ci(data, n_bootstrap=10000, ci=0.95):
"""Bootstrap confidence interval for mean."""
means = []
for _ in range(n_bootstrap):
sample = np.random.choice(data, size=len(data), replace=True)
means.append(np.mean(sample))
lower = np.percentile(means, (1 - ci) / 2 * 100)
upper = np.percentile(means, (1 + ci) / 2 * 100)
return {"mean": np.mean(data), "ci_lower": lower, "ci_upper": upper}
def cohens_h(p1, p2):
"""Cohen's h effect size for two proportions."""
return 2 * np.arcsin(np.sqrt(p1)) - 2 * np.arcsin(np.sqrt(p2))
```
### Reporting Standards
Always include in the paper:
- **Sample sizes**: n=X problems/tasks
- **Number of runs**: K independent runs if applicable
- **Error bars**: Specify standard deviation or standard error
- **Confidence intervals**: 95% CI for key results
- **Significance tests**: p-values for key comparisons
- **Effect sizes**: Cohen's d or h for practical significance
---
## Monitoring (Cron Pattern)
### Cron Prompt Template
For each experiment batch, create a monitoring prompt:
```
Check the status of the [EXPERIMENT_NAME] experiment:
1. Process check: ps aux | grep [PROCESS_PATTERN]
2. Log check: tail -30 [LOG_FILE]
3. Results check: ls [RESULT_DIR]/eval/ (or appropriate result location)
4. If results are available:
- Read the result JSON files
- Report metrics in a table (Borda scores, accuracy, etc.)
- Compute key comparisons between methods
5. If all experiments in this batch are complete:
- git add -A && git commit -m "[COMMIT_MESSAGE]" && git push
- Report final summary
6. Key question: [SPECIFIC ANALYTICAL QUESTION]
If nothing has changed since the last check, respond with [SILENT].
```
### Monitoring Best Practices
1. **Check processes first** — don't read results if the experiment is still running and results are incomplete
2. **Read the log tail** — look for errors, progress indicators, completion messages
3. **Count completed vs expected** — "45/150 problems done" is more useful than "some results exist"
4. **Report in structured tables** — always include key metrics in a table
5. **Answer the key question** — each experiment should have a specific analytical question to answer when done
6. **[SILENT] for no-news** — suppress notifications when nothing has changed
7. **Commit on completion** — every completed batch gets committed with a descriptive message
### Example Monitoring Report
```
## Code Experiments (Haiku 3.5) - COMPLETE
| Strategy | Pass Rate (150 problems) | vs Single |
|----------|------------------------|-----------|
| single_pass | 38.0% | — |
| critique_revise | 35.2% | -2.8pp |
| **autoreason** | **40.0%** | **+2.0pp** |
| best_of_6 | 31.0% | -7.0pp |
Key finding: Autoreason shows +2pp improvement over single pass, while
best-of-6 collapses due to single-public-test selection issue.
Committed: `git commit -m "Add Haiku code results (150 problems, 4 strategies)"`
Next: Run significance tests on these results.
```
---
## Failure Recovery
### Common Failures and Recovery
| Failure | Detection | Recovery |
|---------|-----------|----------|
| **API credit exhaustion** | 402 errors in logs, incomplete results | Top up credits, re-run (skips completed work automatically) |
| **Rate limiting** | 429 errors, slow progress | Add retry logic with exponential backoff |
| **Process crash** | PID gone, log stops mid-problem | Re-run script (resumes from last checkpoint) |
| **Wrong model ID** | Model not found errors | Fix ID (e.g., `claude-opus-4-6` not `claude-opus-4.6`) |
| **Parallel slowdown** | Each experiment taking 2x longer | Reduce parallel experiments to 2-3 max |
| **Security scan blocks** | Commands blocked by security | Use `execute_code` instead of piped `terminal` commands |
| **Delegation failures** | `delegate_task` returns errors | Fall back to doing work directly |
| **Timeout on hard problems** | Process stuck, no log progress | Kill, skip problem, note in results |
| **Dataset path mismatch** | File not found errors | Verify paths before launching |
### Retry Naming Convention
When re-running failed experiments, use a suffix to track rounds:
```
logs/experiment_haiku_0_50.log # Round 1
logs/experiment_haiku_0_50_r2.log # Round 2 (after credit exhaustion)
logs/experiment_haiku_0_50_r3.log # Round 3 (after bug fix)
```
### Pre-Flight Checklist
Before launching any experiment batch:
```
Pre-Flight:
- [ ] API credits sufficient for estimated calls
- [ ] Model IDs correct (test with 1 problem first)
- [ ] Output directory exists and is writable
- [ ] Resume logic works (re-run won't overwrite existing results)
- [ ] Log file path is unique (won't overwrite previous logs)
- [ ] Dataset/task files are accessible
- [ ] Config matches intended experiment
```
---
## Task/Benchmark Design
### Open-Ended Tasks (Subjective Evaluation)
Design tasks that have clear objectives but subjective quality:
```markdown
# Task: [Title]
## Context
[Specific scenario with concrete details: company size, constraints, timeline]
## Deliverable
[Exact format and structure required]
## Requirements
- [Specific, measurable requirements]
- [Not vague — "be comprehensive" is bad, "include exactly 6 sections" is good]
```
### Constrained Tasks (for Testing Scope Effects)
Constrained tasks test whether methods respect scope boundaries. Design with:
- **Fixed facts**: "Use only these N data points, add nothing else"
- **Fixed deliverable**: Specific format (pitch, postmortem, memo — not "improve this")
- **Fixed structure**: "These sections in this order, do not add/remove"
- **Fixed change items**: "Address exactly these N points, nothing else"
**Do NOT use word count as a scope constraint.** Word limits cause false convergence — outputs get rejected for length, not quality. Constrain scope (what to include) not length.
### Example: Good vs Bad Constraints
| Bad Constraint | Why | Good Constraint |
|---------------|-----|-----------------|
| "Max 500 words" | Judges reject for length | "Exactly 4 sections, each with 3 numbered items" |
| "Be concise" | Too vague | "Each prohibition must reference a specific base fact" |
| "Improve this" | Unbounded scope | "Write a 600-word incident postmortem with this exact structure" |
| "Make it better" | No clear criterion | "Address exactly these 3 reviewer concerns" |
---
## Visualization Best Practices
### Setup: SciencePlots + matplotlib
Install SciencePlots for publication-ready defaults:
```bash
pip install SciencePlots matplotlib numpy
```
**Option A: SciencePlots styles** (recommended — handles most defaults automatically):
```python
import matplotlib.pyplot as plt
import scienceplots # registers the styles
# Pick a style:
# 'science' — clean, serif fonts, suitable for most venues
# 'science+ieee' — IEEE-style (good for two-column papers)
# 'science+nature' — Nature-style
# Add 'no-latex' if LaTeX is not installed on the machine generating plots
with plt.style.context(['science', 'no-latex']):
fig, ax = plt.subplots(figsize=(3.5, 2.5)) # single-column width
# ... plot ...
fig.savefig('paper/fig_results.pdf', bbox_inches='tight')
```
**Option B: Manual rcParams** (when you need full control):
```python
import matplotlib.pyplot as plt
plt.rcParams.update({
'font.size': 10,
'font.family': 'serif',
'axes.labelsize': 11,
'axes.titlesize': 11,
'xtick.labelsize': 9,
'ytick.labelsize': 9,
'legend.fontsize': 9,
'figure.figsize': (3.5, 2.5), # single-column default
'figure.dpi': 300,
'savefig.dpi': 300,
'savefig.bbox': 'tight',
'savefig.pad_inches': 0.05,
'axes.linewidth': 0.8,
'lines.linewidth': 1.5,
'lines.markersize': 5,
'axes.grid': True,
'grid.alpha': 0.3,
'grid.linewidth': 0.5,
})
```
### Standard Figure Sizes (Two-Column Format)
| Use Case | figsize | Notes |
|----------|---------|-------|
| Single column | `(3.5, 2.5)` | Fits in one column of two-column layout |
| Double column | `(7.0, 3.0)` | Spans full page width |
| Square (heatmap, confusion matrix) | `(3.5, 3.5)` | Single column |
| Tall single (many rows) | `(3.5, 5.0)` | Use sparingly |
### Colorblind-Safe Palette (Okabe-Ito)
Use this palette for all paper figures. It is distinguishable by people with all common forms of color vision deficiency:
```python
COLORS = {
'blue': '#0072B2',
'orange': '#E69F00',
'green': '#009E73',
'red': '#D55E00',
'purple': '#CC79A7',
'cyan': '#56B4E9',
'yellow': '#F0E442',
'black': '#000000',
}
# As a list for cycling:
COLOR_CYCLE = ['#0072B2', '#D55E00', '#009E73', '#E69F00', '#CC79A7', '#56B4E9']
```
Also differentiate lines by **marker and linestyle**, not just color:
```python
STYLES = [
{'color': '#0072B2', 'marker': 'o', 'linestyle': '-'},
{'color': '#D55E00', 'marker': 's', 'linestyle': '--'},
{'color': '#009E73', 'marker': '^', 'linestyle': '-.'},
{'color': '#E69F00', 'marker': 'D', 'linestyle': ':'},
]
```
### Complete Example: Method Comparison Bar Chart
```python
import matplotlib.pyplot as plt
import numpy as np
try:
import scienceplots
style = ['science', 'no-latex']
except ImportError:
style = 'default'
with plt.style.context(style):
methods = ['Single Pass', 'Critique+Revise', 'Best-of-N', 'Ours']
scores = [73.2, 74.1, 68.5, 77.0]
errors = [2.1, 1.8, 3.2, 1.5]
colors = ['#56B4E9', '#E69F00', '#CC79A7', '#0072B2']
fig, ax = plt.subplots(figsize=(3.5, 2.5))
bars = ax.bar(methods, scores, yerr=errors, capsize=3,
color=colors, edgecolor='black', linewidth=0.5)
# Highlight "Ours"
bars[-1].set_edgecolor('#0072B2')
bars[-1].set_linewidth(1.5)
ax.set_ylabel('Pass Rate (%)')
ax.set_ylim(60, 85)
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
fig.savefig('paper/fig_comparison.pdf', bbox_inches='tight')
```
### Complete Example: Convergence/Trajectory Line Chart
```python
with plt.style.context(style):
fig, ax = plt.subplots(figsize=(3.5, 2.5))
passes = np.arange(1, 16)
ours = [65, 72, 78, 82, 85, 87, 88, 89, 89.5, 90, 90, 90, 90, 90, 90]
baseline = [65, 68, 70, 71, 69, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58]
ax.plot(passes, ours, **STYLES[0], label='Ours', markersize=4)
ax.plot(passes, baseline, **STYLES[1], label='Critique+Revise', markersize=4)
# Mark convergence point
ax.axvline(x=10, color='gray', linestyle=':', alpha=0.5, linewidth=0.8)
ax.annotate('Converged', xy=(10, 90), fontsize=8, ha='center',
xytext=(10, 93), arrowprops=dict(arrowstyle='->', color='gray'))
ax.set_xlabel('Iteration')
ax.set_ylabel('Quality Score')
ax.legend(loc='lower right')
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
fig.savefig('paper/fig_trajectory.pdf', bbox_inches='tight')
```
### Output Rules
- **Always save as PDF**: `fig.savefig('fig.pdf')` — vector graphics, sharp at any zoom
- **Never save as PNG** for paper figures — raster PNGs look blurry when printed/zoomed
- **Exception**: Screenshots, photographs, or pixel-art visualizations → PNG at 600 DPI
- **Verify grayscale**: Print to grayscale PDF and check all information is still visible
### Chart Types for Common Comparisons
| Comparison Type | Chart | Notes |
|----------------|-------|-------|
| Method vs method | Grouped bar chart | Include error bars |
| Across model sizes | Line chart with CI bands | Log scale for model size axis |
| Ablation study | Stacked/grouped bar | Highlight removed component |
| Trajectory/convergence | Line chart over iterations | Show winner per iteration |
| Per-task breakdown | Heatmap or grouped bar | Show variance across tasks |

68
skills/research/ml-paper-writing/references/reviewer-guidelines.md → skills/research/research-paper-writing/references/reviewer-guidelines.md
View File

@@ -105,7 +105,7 @@ Reviewers are explicitly instructed to:
- Penalizing authors for honest limitation acknowledgment
- Rejecting for missing citations to reviewer's own work
### Timeline (NeurIPS 2025)
### Timeline (NeurIPS 2025 — verify dates for current year)
- Bidding: May 17-21
- Reviewing period: May 29 - July 2
@@ -113,6 +113,8 @@ Reviewers are explicitly instructed to:
- Discussion period: July 31 - August 13
- Final notifications: September 18
> **Note**: These dates are from the 2025 cycle. Always check the current year's call for papers at the venue website.
---
## ICML Reviewer Guidelines
@@ -198,6 +200,70 @@ ACL has a dedicated ethics review process for:
---
## AAAI Reviewer Guidelines
### Evaluation Criteria
AAAI reviewers evaluate along similar axes to NeurIPS/ICML but with some differences:
| Criterion | Weight | Notes |
|-----------|--------|-------|
| **Technical quality** | High | Soundness of approach, correctness of results |
| **Significance** | High | Importance of the problem and contribution |
| **Novelty** | Medium-High | New ideas, methods, or insights |
| **Clarity** | Medium | Clear writing, well-organized presentation |
| **Reproducibility** | Medium | Sufficient detail to reproduce results |
### AAAI-Specific Considerations
- **Broader AI scope**: AAAI covers all of AI, not just ML. Papers on planning, reasoning, knowledge representation, NLP, vision, robotics, and multi-agent systems are all in scope. Reviewers may not be deep ML specialists.
- **Formatting strictness**: AAAI reviewers are instructed to flag formatting violations. Non-compliant papers may be desk-rejected before review.
- **Application papers**: AAAI is more receptive to application-focused work than NeurIPS/ICML. Framing a strong application contribution is viable.
- **Senior Program Committee**: AAAI uses SPCs (Senior Program Committee members) who mediate between reviewers and make accept/reject recommendations.
### Scoring (AAAI Scale)
- **Strong Accept**: Clearly above threshold, excellent contribution
- **Accept**: Above threshold, good contribution with minor issues
- **Weak Accept**: Borderline, merits outweigh concerns
- **Weak Reject**: Borderline, concerns outweigh merits
- **Reject**: Below threshold, significant issues
- **Strong Reject**: Well below threshold
---
## COLM Reviewer Guidelines
### Evaluation Criteria
COLM reviews focus on relevance to language modeling in addition to standard criteria:
| Criterion | Weight | Notes |
|-----------|--------|-------|
| **Relevance** | High | Must be relevant to language modeling community |
| **Technical quality** | High | Sound methodology, well-supported claims |
| **Novelty** | Medium-High | New insights about language models |
| **Clarity** | Medium | Clear presentation, reproducible |
| **Significance** | Medium-High | Impact on LM research and practice |
### COLM-Specific Considerations
- **Language model focus**: Reviewers will assess whether the contribution advances understanding of language models. General ML contributions need explicit LM framing.
- **Newer venue norms**: COLM is newer than NeurIPS/ICML, so reviewer calibration varies more. Write more defensively — anticipate a wider range of reviewer expertise.
- **ICLR-derived process**: Review process is modeled on ICLR (open reviews, author response period, discussion among reviewers).
- **Broad interpretation of "language modeling"**: Includes training, evaluation, alignment, safety, efficiency, applications, theory, multimodality (if language is central), and social impact of LMs.
### Scoring
COLM uses an ICLR-style scoring system:
- **8-10**: Strong accept (top papers)
- **6-7**: Weak accept (solid contribution)
- **5**: Borderline
- **3-4**: Weak reject (below threshold)
- **1-2**: Strong reject
---
## What Makes Reviews Strong
### Following Daniel Dennett's Rules

0
skills/research/ml-paper-writing/references/sources.md → skills/research/research-paper-writing/references/sources.md
View File

2
skills/research/ml-paper-writing/references/writing-guide.md → skills/research/research-paper-writing/references/writing-guide.md
View File

@@ -225,8 +225,6 @@ Provide context before asking the reader to consider anything new. This applies
---
---
## Micro-Level Writing Tips
### From Ethan Perez (Anthropic)

0
skills/research/ml-paper-writing/templates/README.md → skills/research/research-paper-writing/templates/README.md
View File

0
skills/research/ml-paper-writing/templates/aaai2026/README.md → skills/research/research-paper-writing/templates/aaai2026/README.md
View File

0
skills/research/ml-paper-writing/templates/aaai2026/aaai2026-unified-supp.tex → skills/research/research-paper-writing/templates/aaai2026/aaai2026-unified-supp.tex
View File

0
skills/research/ml-paper-writing/templates/aaai2026/aaai2026-unified-template.tex → skills/research/research-paper-writing/templates/aaai2026/aaai2026-unified-template.tex
View File

0
skills/research/ml-paper-writing/templates/aaai2026/aaai2026.bib → skills/research/research-paper-writing/templates/aaai2026/aaai2026.bib
View File

0
skills/research/ml-paper-writing/templates/aaai2026/aaai2026.bst → skills/research/research-paper-writing/templates/aaai2026/aaai2026.bst
View File

0
skills/research/ml-paper-writing/templates/aaai2026/aaai2026.sty → skills/research/research-paper-writing/templates/aaai2026/aaai2026.sty
View File

0
skills/research/ml-paper-writing/templates/acl/README.md → skills/research/research-paper-writing/templates/acl/README.md
View File

0
skills/research/ml-paper-writing/templates/acl/acl.sty → skills/research/research-paper-writing/templates/acl/acl.sty
View File

0
skills/research/ml-paper-writing/templates/acl/acl_latex.tex → skills/research/research-paper-writing/templates/acl/acl_latex.tex
View File

0
skills/research/ml-paper-writing/templates/acl/acl_lualatex.tex → skills/research/research-paper-writing/templates/acl/acl_lualatex.tex
View File

0
skills/research/ml-paper-writing/templates/acl/acl_natbib.bst → skills/research/research-paper-writing/templates/acl/acl_natbib.bst
View File

0
skills/research/ml-paper-writing/templates/acl/anthology.bib.txt → skills/research/research-paper-writing/templates/acl/anthology.bib.txt
View File

0
skills/research/ml-paper-writing/templates/acl/custom.bib → skills/research/research-paper-writing/templates/acl/custom.bib
View File

0
skills/research/ml-paper-writing/templates/acl/formatting.md → skills/research/research-paper-writing/templates/acl/formatting.md
View File

0
skills/research/ml-paper-writing/templates/colm2025/README.md → skills/research/research-paper-writing/templates/colm2025/README.md
View File

0
skills/research/ml-paper-writing/templates/colm2025/colm2025_conference.bib → skills/research/research-paper-writing/templates/colm2025/colm2025_conference.bib
View File

0
skills/research/ml-paper-writing/templates/colm2025/colm2025_conference.bst → skills/research/research-paper-writing/templates/colm2025/colm2025_conference.bst
View File

0
skills/research/ml-paper-writing/templates/colm2025/colm2025_conference.pdf → skills/research/research-paper-writing/templates/colm2025/colm2025_conference.pdf
View File

0
skills/research/ml-paper-writing/templates/colm2025/colm2025_conference.sty → skills/research/research-paper-writing/templates/colm2025/colm2025_conference.sty
View File

0
skills/research/ml-paper-writing/templates/colm2025/colm2025_conference.tex → skills/research/research-paper-writing/templates/colm2025/colm2025_conference.tex
View File

0
skills/research/ml-paper-writing/templates/colm2025/fancyhdr.sty → skills/research/research-paper-writing/templates/colm2025/fancyhdr.sty
View File

0
skills/research/ml-paper-writing/templates/colm2025/math_commands.tex → skills/research/research-paper-writing/templates/colm2025/math_commands.tex
View File

0
skills/research/ml-paper-writing/templates/colm2025/natbib.sty → skills/research/research-paper-writing/templates/colm2025/natbib.sty
View File

0
skills/research/ml-paper-writing/templates/iclr2026/fancyhdr.sty → skills/research/research-paper-writing/templates/iclr2026/fancyhdr.sty
View File

0
skills/research/ml-paper-writing/templates/iclr2026/iclr2026_conference.bib → skills/research/research-paper-writing/templates/iclr2026/iclr2026_conference.bib
View File

0
skills/research/ml-paper-writing/templates/iclr2026/iclr2026_conference.bst → skills/research/research-paper-writing/templates/iclr2026/iclr2026_conference.bst
View File

0
skills/research/ml-paper-writing/templates/iclr2026/iclr2026_conference.pdf → skills/research/research-paper-writing/templates/iclr2026/iclr2026_conference.pdf
View File

0
skills/research/ml-paper-writing/templates/iclr2026/iclr2026_conference.sty → skills/research/research-paper-writing/templates/iclr2026/iclr2026_conference.sty
View File

0
skills/research/ml-paper-writing/templates/iclr2026/iclr2026_conference.tex → skills/research/research-paper-writing/templates/iclr2026/iclr2026_conference.tex
View File

0
skills/research/ml-paper-writing/templates/iclr2026/math_commands.tex → skills/research/research-paper-writing/templates/iclr2026/math_commands.tex
View File

0
skills/research/ml-paper-writing/templates/iclr2026/natbib.sty → skills/research/research-paper-writing/templates/iclr2026/natbib.sty
View File

0
skills/research/ml-paper-writing/templates/icml2026/algorithm.sty → skills/research/research-paper-writing/templates/icml2026/algorithm.sty
View File

0
skills/research/ml-paper-writing/templates/icml2026/algorithmic.sty → skills/research/research-paper-writing/templates/icml2026/algorithmic.sty
View File

0
skills/research/ml-paper-writing/templates/icml2026/example_paper.bib → skills/research/research-paper-writing/templates/icml2026/example_paper.bib
View File

0
skills/research/ml-paper-writing/templates/icml2026/example_paper.pdf → skills/research/research-paper-writing/templates/icml2026/example_paper.pdf
View File

0
skills/research/ml-paper-writing/templates/icml2026/example_paper.tex → skills/research/research-paper-writing/templates/icml2026/example_paper.tex
View File

0
skills/research/ml-paper-writing/templates/icml2026/fancyhdr.sty → skills/research/research-paper-writing/templates/icml2026/fancyhdr.sty
View File

0
skills/research/ml-paper-writing/templates/icml2026/icml2026.bst → skills/research/research-paper-writing/templates/icml2026/icml2026.bst
View File

0
skills/research/ml-paper-writing/templates/icml2026/icml2026.sty → skills/research/research-paper-writing/templates/icml2026/icml2026.sty
View File

0
skills/research/ml-paper-writing/templates/icml2026/icml_numpapers.pdf → skills/research/research-paper-writing/templates/icml2026/icml_numpapers.pdf
View File

0
skills/research/ml-paper-writing/templates/neurips2025/Makefile → skills/research/research-paper-writing/templates/neurips2025/Makefile
View File

0
skills/research/ml-paper-writing/templates/neurips2025/extra_pkgs.tex → skills/research/research-paper-writing/templates/neurips2025/extra_pkgs.tex
View File

0
skills/research/ml-paper-writing/templates/neurips2025/main.tex → skills/research/research-paper-writing/templates/neurips2025/main.tex
View File

0
skills/research/ml-paper-writing/templates/neurips2025/neurips.sty → skills/research/research-paper-writing/templates/neurips2025/neurips.sty
View File