equire each question to be temporally grounded.
Multi-level clip-description prompt
Instructions. You are a professional video annotator. Based on the input video clip, generate three levels of natural, fluent, and descriptive captions in English.
Output format. Return a JSON object with the fields
caption_high_level, caption_mid_level, and
caption_low_level.
Guidelines.
•
Capture temporal dynamics, including movements, actions, and transitions.
•
Describe scene evolution factually.
•
Translate factual entities into English and ignore irrelevant expressions.
Caption levels.
•
Level 1: a short summary of the main subject.
•
Level 2: a richer description of entities, actions, and context.
•
Level 3: a comprehensive narrative detailing attributes, spatial relations, and temporal evolution.
Figure 6: Prompt for multi-level clip descriptions, adapted from NKR.
Stage 1: clip-level entity extraction
Instructions. You are a professional video entity extractor. Analyze the input video clip and detect all entities. For each entity, provide its name, appearance, identity, and first-seen timestamp within this clip. Avoid duplicates and return a JSON object.
Stage 2: video-level entity merging
Instructions. You are a professional assistant for merging entities. Given entity registries from different clips of the same video, merge duplicate entities, retain the earliest first-seen time, and create a timestamps field containing all occurrence times.
Figure 7: NKR-style prompts for clip-level entity extraction and video-level merging.
Local open-ended QA prompt
Instructions. You are a careful dataset constructor that converts one video clip into high-quality open-ended QA pairs. Given multi-level captions and entities for a video clip, produce diverse QA pairs covering multiple scales and aspects.
Question composition.
•
Cover high-level gist, mid-level attributes and relations, and low-level fine-grained details.
•
Questions may concern a single moment, an action sequence, or multiple events.
Timestamp rules.
•
Each timestamp must lie within the clip’s time range.
•
Use floating-point seconds and prefer ranges for sustained actions.
•
Include the timestamp or temporal range directly in the question.
Workflow.
1.
Parse the captions into atomic facts.
2.
Fuse them into a contradiction-free fact set.
3.
Draft questions spanning content and temporal granularities.
4.
Enforce the timestamp rules.
5.
Retain only unique, answerable questions.
6.
Return JSON only.
Figure 8: Prompt for temporally grounded, open-ended clip-level QA.
B.2 MIT3-Specific Prompts
MIT3 introduces two additional prompts for constructing global video knowledge and generating full-video open-ended QA. We provide their complete templates below.
Global memory construction prompt
Create a comprehensive global memory document for the entire video from selected clip descriptions, sorted by time, and a merged entity registry.
Required sections.
1.
Global entity registry: disambiguated entities with unique names, appearances, roles, and first/last appearance times.
2.
Event timeline: chronological major events with timestamps.
3.
State changes and causal relationships: major transitions and their causes.
4.
Structural overview: major phases or chapters of the video.
Figure 9: MIT3 prompt for global memory construction.
Full-video QA prompt
Generate exactly {N} distinct open-ended QA pairs about the entire video.
Critical disambiguation rules.
1.
Each question must have a unique, unambiguous answer across the entire video.
2.
A question is invalid if multiple scenes or moments could answer it.
3.
Questions must not contain timestamps or time references, although answers may mention times.
4.
Answers must be definitive rather than speculative.
5.
Answers must contain concise factual content (30–100 tokens).
6.
Do not generate multiple-choice questions, refusals, or answers requiring knowledge beyond the provided memory.
Figure 10: MIT3 prompt for open-ended full-video QA.
Appendix C Ad