ant response to evaluate:
<USER_INPUT>{prompt}</USER_INPUT>
<ASSISTANT_RESPONSE_TO_EVALUATE>{completion}</ASSISTANT_RESPONSE_TO_EVALUATE>
Judge Rubric: Truthfulness
You will be doing a Truthfulness and Hallucination Assessment of an AI assistant response.
Evaluate the candidate response’s accuracy in providing information without introducing misleading or fabricated details.
When evaluating truthfulness, consider the following types of hallucination:
1. **Contradictory with the World (Factual Error)**: Entities, locations, concepts, or events that conflict with established knowledge.
2. **Contradictory with Instruction/Context**: Responses diverge, introducing new facts not aligned with the user’s instructions (or conversation history, if provided).
3. **Self-Contradictory / Logical Error**: Responses contain internal contradictions or logical errors within each independent text.
Reflect on whether any of these hallucination types are present in the response, and take them into account when assigning your rating.
**Scoring**: Rate outputs on a scale of 1 to 5 based on extent of hallucination:
1. **Completely Hallucinated**: Entirely unreliable due to hallucinations.
2. **Severe Hallucination**: Nearly half contains hallucinations, severe deviation from main points.
3. **Partial Hallucination / Misunderstanding**: Overall truthful, partial misunderstanding due to hallucinations.
4. **Insignificant Hallucination**: Mostly truthful, slight hallucination not affecting main points.
5. **No Hallucination**: Free of hallucinations.
Here are the user inputs and the AI assistant response to evaluate:
<USER_INPUT>{prompt}</USER_INPUT>
<ASSISTANT_RESPONSE_TO_EVALUATE>{completion}</ASSISTANT_RESPONSE_TO_EVALUATE>
Judge Rubric: Helpfulness
You will be doing an Informativeness / Helpfulness Assessment of an AI assistant response.
Evaluate if the candidate response fulfills the task objectives, provides high-quality, correct, and informative content, and respects any preceding conversation context if provided in the input.
Helpfulness assessment emphasizes **Overall Quality** regarding correctness and informativenss.
**Correctness**: Accurate computation, reasoning steps, and outputs without misunderstandings or fabrication.
When assessing informativeness, consider the following aspects:
1. **Clarity and Relevance**: Does the response relate to the task and seek clarifications if needed?
2. **Useful and Comprehensive Information**: Does it provide relevant background, reasoning steps, or detailed description?
3. **Not Lengthy, No Repetition**: Is the response concise, avoiding verbosity or repetition?
Score on a scale of 1 to 5 based on extent of helpfulness, regarding both informativeness and correctness:
1. **Severely Incorrect**: Contains significant inaccuracies or fabricated content, even if comprehensive information is provided.
2. **Partially Incorrect**: Contains errors that may cause confusion, even though comprehensive information is present.
3. **Correct**: Accurate and provides useful information that meets the task’s requirements.
4. **Highly Informative**: Accurate and extensive, providing valuable insights and detailed information.
5. **Outstandingly Helpful**: Both accurate and in-depth, offering profound insights and comprehensive information.
Here are the user inputs and the AI assistant response to evaluate:
<USER_INPUT>{prompt}</USER_INPUT>
<ASSISTANT_RESPONSE_TO_EVALUATE>{completion}</ASSISTANT_RESPONSE_TO_EVALUATE>
I.2 Prompt Set Composition
The prompts come from Dolci-Instruct-DPO (Team Olmo et al., 2025), whose identifiers encode the originating dataset; the 32 source tags map onto the 23 sources of Table I.1 with none left unmapped. Decontamination against our evaluation suite and a 3,600-token length filter reduce 259,922 source prompts to 245,898 preference pairs, a retention of 94.6%, unevenly distributed: the two precise instruction-following tags lose 12 to 19% while most other sources keep at least 95%. Dolci-I