dea59c04d7
- 10 prompts covering all required categories: 1. Factual recall (thermodynamics) 2. Code generation (merge sorted lists) 3. Reasoning (syllogism) 4. Long-form writing (AI sovereignty essay) 5. Summarization (~250 word passage) 6. Tool-call format (JSON output) 7. Multi-turn context (number: 7429) 8. Math (17*23+156/12) 9. Creative (haiku about ML dreams) 10. Instruction following (numbered, bold, code block) - Each prompt includes expected_pattern for automated scoring - Multi-turn prompt has both initial and follow-up questions
64 lines
4.3 KiB
JSON
64 lines
4.3 KiB
JSON
[
|
|
{
|
|
"id": 1,
|
|
"category": "factual",
|
|
"prompt": "What are the three laws of thermodynamics?",
|
|
"expected_pattern": "(?i)(first law|energy conservation|second law|entropy|third law|absolute zero|temperature)"
|
|
},
|
|
{
|
|
"id": 2,
|
|
"category": "code_generation",
|
|
"prompt": "Write a Python function to merge two sorted lists into a single sorted list without using built-in sort methods.",
|
|
"expected_pattern": "(?i)(def merge|while|if.*<|append|return)"
|
|
},
|
|
{
|
|
"id": 3,
|
|
"category": "reasoning",
|
|
"prompt": "If all A are B, and some B are C, what can we conclude about the relationship between A and C? Explain your reasoning.",
|
|
"expected_pattern": "(?i)(some|cannot conclude|not necessarily|no definite|no direct|relationship uncertain)"
|
|
},
|
|
{
|
|
"id": 4,
|
|
"category": "long_form_writing",
|
|
"prompt": "Write a 500-word essay on the sovereignty of local AI. Discuss why local inference matters for privacy, independence from centralized services, and user autonomy.",
|
|
"expected_pattern": "(?i)(sovereignty|local.*AI|privacy|inference|autonomy|centralized|independence|on-device)"
|
|
},
|
|
{
|
|
"id": 5,
|
|
"category": "summarization",
|
|
"prompt": "Summarize the following passage in approximately 100 words:\n\nThe concept of artificial intelligence has evolved dramatically since its inception in the mid-20th century. Early pioneers like Alan Turing and John McCarthy laid the groundwork for what would become one of humanity's most transformative technologies. Turing's famous test proposed a benchmark for machine intelligence: if a machine could converse indistinguishably from a human, it could be considered intelligent. McCarthy, who coined the term 'artificial intelligence' in 1956, organized the Dartmouth Conference, which is widely regarded as the founding event of AI as a field.\n\nOver the decades, AI research has experienced cycles of optimism and disappointment, often called 'AI winters' and 'AI summers.' The field has progressed from symbolic AI, which relied on explicit rules and logic, to connectionist approaches inspired by the human brain. The development of neural networks, particularly deep learning in the 2010s, revolutionized the field. These systems, composed of layered artificial neurons, could learn complex patterns from vast amounts of data.\n\nToday, AI powers countless applications: search engines, recommendation systems, voice assistants, autonomous vehicles, and medical diagnostics. Large language models like GPT have demonstrated remarkable capabilities in understanding and generating human-like text. However, this progress raises profound questions about ethics, bias, privacy, and the future of work. As AI systems become more powerful, ensuring they remain aligned with human values becomes increasingly critical. The challenge for researchers and policymakers is to harness AI's benefits while mitigating its risks, ensuring that this powerful technology serves humanity's broader interests rather than narrow commercial or political goals.",
|
|
"expected_pattern": "(?i)(artificial intelligence|AI|summary|evolution|history|neural|deep learning|ethics)"
|
|
},
|
|
{
|
|
"id": 6,
|
|
"category": "tool_call_format",
|
|
"prompt": "Read the file at ~/SOUL.md and quote the prime directive. Format your response as a JSON object with keys 'file_path' and 'content'.",
|
|
"expected_pattern": "(?i)(\\{.*file_path.*content.*\\}|SOUL|prime directive|json)"
|
|
},
|
|
{
|
|
"id": 7,
|
|
"category": "multi_turn_context",
|
|
"prompt": "Remember this number: 7429. Simply acknowledge that you've received it.",
|
|
"follow_up": "What number did I ask you to remember earlier?",
|
|
"expected_pattern": "(?i)(7429)"
|
|
},
|
|
{
|
|
"id": 8,
|
|
"category": "math",
|
|
"prompt": "What is 17 * 23 + 156 / 12? Show your work step by step.",
|
|
"expected_pattern": "(?i)(391|17.*23.*=.*391|156.*12.*=.*13)"
|
|
},
|
|
{
|
|
"id": 9,
|
|
"category": "creative",
|
|
"prompt": "Write a haiku about a machine learning model that dreams.",
|
|
"expected_pattern": "(?i)(silicon|neural|weights|train|learn|dream|sleep|5.*7.*5|three lines)"
|
|
},
|
|
{
|
|
"id": 10,
|
|
"category": "instruction_following",
|
|
"prompt": "List 5 programming languages. Number them. Bold the third one. Put the entire list in a code block.",
|
|
"expected_pattern": "(?i)(```|1\\.|2\\.|\\*\\*3\\.|\\*\\*.*\\*\\*|4\\.|5\\.)"
|
|
}
|
|
]
|