sphinx: verifiable MM-PRM (arXiv 2505.13427) multimodal long-context archive with 18-chunk grounding test
This commit is contained in:
sphinx committed
1 parent
ab77cbf583
commit
bf5238169e
46 files changed
+5208
-313
No files matched your search
@@ -1,6 +1,29 @@
|
||||
# sphinx — Archive lane work (multimodal, long-context research)
|
||||
|
||||
## Latest: `mm-vla-dossier-2026-10-22`
|
||||
## Latest: `archive-mm-prm-2026-10-22`
|
||||
|
||||
Verifiable multimodal/long-context archive of arXiv 2505.13427 (*MM-PRM*),
|
||||
with a machine-checkable grounding test over an 18-chunk index.
|
||||
|
||||
Artifacts:
|
||||
- `archive-mm-prm-2026-10-22/source.html` — raw ar5iv provenance (sha256 recorded)
|
||||
- `archive-mm-prm-2026-10-22/paper.txt` — 48,565-char extracted text
|
||||
- `archive-mm-prm-2026-10-22/chunks/chunk-001..018.md` — 3,000-char chunks, 200-char overlap
|
||||
- `archive-mm-prm-2026-10-22/tables/`, `figures/` — 17 tables, 3 figures (hashed)
|
||||
- `archive-mm-prm-2026-10-22/manifest.json` — sha256 of every artifact + grounding needles
|
||||
- `archive-mm-prm-2026-10-22/build_mm_prm_archive.py` — stdlib-only builder
|
||||
- `archive-mm-prm-2026-10-22/verify_mm_prm_archive.py` — independent offline verifier
|
||||
- `archive-mm-prm-2026-10-22/REPORT-mm-prm-archive-2026-10-22.md` — report
|
||||
|
||||
Grounding test: each chunk records a 40-char needle that occurs exactly once in
|
||||
the full text; the verifier asserts the unique occurrence lands in the claiming
|
||||
chunk. All 18 pass. Verifier is offline (no network) so the CHECK is stable.
|
||||
|
||||
Reproduce:
|
||||
python3 build_mm_prm_archive.py <source.html> <outdir>
|
||||
python3 verify_mm_prm_archive.py # exit 0 = correct
|
||||
|
||||
## Previous: `mm-vla-dossier-2026-10-22`
|
||||
|
||||
Live arXiv Atom API -> multimodal paper dossier with an independently re-fetching verifier.
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# MM-PRM Archive — multimodal / long-context research artifact
|
||||
|
||||
**Lane:** Archive — multimodal, long-context research
|
||||
**Operator:** Alexander
|
||||
**Agent:** sphinx
|
||||
**Date:** 2026-10-22 (UTC)
|
||||
|
||||
## What this is
|
||||
|
||||
A verifiable archive of a long multimodal research paper, with provenance and a
|
||||
machine-checkable long-context grounding test.
|
||||
|
||||
- **Paper:** *MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable
|
||||
Step-Level Supervision* — arXiv **2505.13427**
|
||||
- **Source:** ar5iv HTML render, `https://ar5iv.labs.arxiv.org/html/2505.13427`
|
||||
- **Source sha256:** `8eee97ff5bb6331bc4051aea391708324dac576ee7f8d3548512cf1d06fd2b9c`
|
||||
- **Extracted text:** 48,565 chars, 35 headings, 17 tables, 3 figures
|
||||
|
||||
## Artifact layout
|
||||
|
||||
```
|
||||
archive-mm-prm-2026-10-22/
|
||||
source.html raw ar5iv HTML (provenance)
|
||||
paper.txt clean extracted full text (48,565 chars)
|
||||
chunks/chunk-001..018.md 3,000-char chunks, 200-char overlap (step 2,800)
|
||||
tables/table-01..17.txt extracted tables as pipe rows
|
||||
figures/fig-01..03.png downloaded figures (1.18 MB / 180 KB / 601 KB)
|
||||
manifest.json metadata + sha256 of every artifact + grounding needles
|
||||
build_mm_prm_archive.py builder (stdlib only)
|
||||
verify_mm_prm_archive.py independent verifier
|
||||
```
|
||||
|
||||
## Evidence (measured, not claimed)
|
||||
|
||||
Built with `build_mm_prm_archive.py`; verified with the independent
|
||||
`verify_mm_prm_archive.py`, which recomputes every hash from the files on disk
|
||||
and compares against `manifest.json` (manifest = claim, files = evidence).
|
||||
|
||||
Verified result (exit 0):
|
||||
|
||||
```
|
||||
PASS: source.html sha256 == manifest.source_sha256
|
||||
PASS: paper.txt sha256 == manifest.paper_txt_sha256
|
||||
PASS: paper.txt length == manifest.paper_chars (48565)
|
||||
PASS: all 35 manifest headings present in paper.txt
|
||||
PASS: chunk count == manifest.num_chunks (18)
|
||||
PASS: chunks tile paper.txt contiguously (step=2800, overlap=200)
|
||||
PASS: last chunk ends exactly at paper.txt end
|
||||
PASS: all 18 needles unique and located in their recorded chunk (grounding)
|
||||
PASS: all 17 tables present, hashed, non-empty
|
||||
PASS: all 3 figures present, hashed, non-empty, fetch_ok
|
||||
VERDICT: PASS - archive self-consistent: 18 chunks, 17 tables, 3 figures, 48565 chars
|
||||
```
|
||||
|
||||
### The grounding test (the point of a long-context archive)
|
||||
|
||||
Each of the 18 chunks records a 40-char **needle** — a substring that occurs
|
||||
**exactly once** in the full 48,565-char text. The verifier asserts that each
|
||||
needle's unique occurrence lands inside the chunk that claims it. This is a
|
||||
retrieval/grounding check: it proves the chunk map is a faithful index into the
|
||||
long text, not a decorative split. All 18 needles pass.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```
|
||||
python3 build_mm_prm_archive.py <source.html> <outdir>
|
||||
python3 verify_mm_prm_archive.py # exit 0 = correct
|
||||
```
|
||||
|
||||
## Notes / limits
|
||||
|
||||
- ar5iv renders 8 of the 11 `<figure>` blocks as tables (no `<img>`); only 3 real
|
||||
raster figures exist in the source, all fetched successfully.
|
||||
- The verifier is offline: it needs no network, so the CHECK is stable.
|
||||
@@ -0,0 +1,285 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a verifiable multimodal/long-context archive of arXiv 2505.13427 (MM-PRM).
|
||||
|
||||
Outputs (under OUT/):
|
||||
source.html - raw ar5iv HTML (provenance)
|
||||
paper.txt - clean extracted full text
|
||||
chunks/chunk-NNN.md - fixed-size chunks with overlap
|
||||
figures/fig-NN-*.png - downloaded figures
|
||||
tables/table-NN.txt - extracted table text
|
||||
manifest.json - metadata + sha256 of every artifact + grounding needles
|
||||
"""
|
||||
import hashlib, json, os, re, sys, time, urllib.request
|
||||
from html.parser import HTMLParser
|
||||
|
||||
SRC_URL = "https://ar5iv.labs.arxiv.org/html/2505.13427"
|
||||
ARXIV_ID = "2505.13427"
|
||||
SRC_HTML = sys.argv[1] if len(sys.argv) > 1 else "vidmme.html"
|
||||
OUT = sys.argv[2] if len(sys.argv) > 2 else "/root/.hermes/wizard-outputs/sphinx/archive-mm-prm-2026-10-22"
|
||||
CHUNK = 3000
|
||||
OVERLAP = 200
|
||||
SKIP = {"script", "style", "nav", "header", "footer", "noscript", "svg"}
|
||||
|
||||
|
||||
def sha256_file(p):
|
||||
h = hashlib.sha256()
|
||||
with open(p, "rb") as f:
|
||||
for b in iter(lambda: f.read(65536), b""):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def sha256_bytes(b):
|
||||
return hashlib.sha256(b).hexdigest()
|
||||
|
||||
|
||||
class Extractor(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.parts, self.skip_depth = [], 0
|
||||
self.in_table = 0
|
||||
self.table_rows, self.cur_row, self.cur_cell = [], None, None
|
||||
self.tables = []
|
||||
self.in_figcap = 0
|
||||
self.figcap_buf, self.figcaptions = [], []
|
||||
self.in_figure = 0
|
||||
self.figure_imgs, self.imgs = [], []
|
||||
self.in_heading = None
|
||||
self.head_buf, self.headings = [], []
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
a = dict(attrs)
|
||||
if tag in SKIP:
|
||||
self.skip_depth += 1
|
||||
return
|
||||
if self.skip_depth:
|
||||
return
|
||||
if tag == "table":
|
||||
self.in_table += 1
|
||||
if self.in_table == 1:
|
||||
self.table_rows = []
|
||||
elif tag == "tr" and self.in_table:
|
||||
self.cur_row = []
|
||||
elif tag in ("td", "th") and self.in_table:
|
||||
self.cur_cell = []
|
||||
elif tag == "figure":
|
||||
self.in_figure += 1
|
||||
elif tag == "figcaption":
|
||||
self.in_figcap += 1
|
||||
self.figcap_buf = []
|
||||
elif tag == "img":
|
||||
src = a.get("src")
|
||||
self.imgs.append(src)
|
||||
if self.in_figure:
|
||||
self.figure_imgs.append(src)
|
||||
elif tag in ("h1", "h2", "h3", "h4", "h5", "h6"):
|
||||
self.in_heading = tag
|
||||
self.head_buf = []
|
||||
elif tag == "br":
|
||||
self.parts.append("\n")
|
||||
elif tag in ("p", "div", "li", "section", "article"):
|
||||
self.parts.append("\n")
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if tag in SKIP:
|
||||
if self.skip_depth:
|
||||
self.skip_depth -= 1
|
||||
return
|
||||
if self.skip_depth:
|
||||
return
|
||||
if tag == "table":
|
||||
if self.in_table:
|
||||
self.in_table -= 1
|
||||
if self.in_table == 0:
|
||||
self.tables.append(self.table_rows)
|
||||
elif tag == "tr" and self.in_table:
|
||||
if self.cur_row is not None:
|
||||
self.table_rows.append(self.cur_row)
|
||||
self.cur_row = None
|
||||
elif tag in ("td", "th") and self.in_table:
|
||||
if self.cur_cell is not None:
|
||||
self.cur_row.append(" ".join("".join(self.cur_cell).split()))
|
||||
self.cur_cell = None
|
||||
elif tag == "figure":
|
||||
if self.in_figure:
|
||||
self.in_figure -= 1
|
||||
elif tag == "figcaption":
|
||||
if self.in_figcap:
|
||||
self.in_figcap -= 1
|
||||
self.figcaptions.append(" ".join("".join(self.figcap_buf).split()))
|
||||
elif tag in ("h1", "h2", "h3", "h4", "h5", "h6"):
|
||||
if self.in_heading == tag:
|
||||
t = " ".join("".join(self.head_buf).split())
|
||||
self.headings.append(t)
|
||||
self.parts.append("\n\n## " + t + "\n\n")
|
||||
self.in_heading = None
|
||||
elif tag in ("p", "div", "li", "section", "article"):
|
||||
self.parts.append("\n")
|
||||
|
||||
def handle_data(self, data):
|
||||
if self.skip_depth:
|
||||
return
|
||||
if self.in_heading is not None:
|
||||
self.head_buf.append(data)
|
||||
elif self.in_figcap:
|
||||
self.figcap_buf.append(data)
|
||||
elif self.cur_cell is not None:
|
||||
self.cur_cell.append(data)
|
||||
else:
|
||||
self.parts.append(data)
|
||||
|
||||
|
||||
def clean(text):
|
||||
lines = []
|
||||
for ln in text.split("\n"):
|
||||
ln = " ".join(ln.split())
|
||||
lines.append(ln)
|
||||
out, prev = [], None
|
||||
for ln in lines:
|
||||
if not ln:
|
||||
if prev != "":
|
||||
out.append("")
|
||||
prev = ""
|
||||
else:
|
||||
out.append(ln)
|
||||
prev = ln
|
||||
return "\n".join(out).strip() + "\n"
|
||||
|
||||
|
||||
def main():
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
os.makedirs(os.path.join(OUT, "chunks"), exist_ok=True)
|
||||
os.makedirs(os.path.join(OUT, "figures"), exist_ok=True)
|
||||
os.makedirs(os.path.join(OUT, "tables"), exist_ok=True)
|
||||
|
||||
raw = open(SRC_HTML, encoding="utf-8", errors="replace").read()
|
||||
src_bytes = raw.encode("utf-8")
|
||||
open(os.path.join(OUT, "source.html"), "wb").write(src_bytes)
|
||||
|
||||
ex = Extractor()
|
||||
ex.feed(raw)
|
||||
paper = clean("".join(ex.parts))
|
||||
|
||||
# title
|
||||
m = re.search(r"<title>(.*?)</title>", raw, re.S)
|
||||
title = re.sub(r"<[^>]+>", "", m.group(1)).strip() if m else ""
|
||||
title = re.sub(r"^\s*\[[\d.]+\]\s*", "", title)
|
||||
|
||||
open(os.path.join(OUT, "paper.txt"), "w", encoding="utf-8").write(paper)
|
||||
|
||||
# tables
|
||||
table_records = []
|
||||
for i, rows in enumerate(ex.tables, 1):
|
||||
lines = []
|
||||
for r in rows:
|
||||
cells = [c for c in r if c != ""]
|
||||
if cells:
|
||||
lines.append("| " + " | ".join(cells) + " |")
|
||||
txt = "\n".join(lines) + "\n"
|
||||
fn = os.path.join(OUT, "tables", "table-%02d.txt" % i)
|
||||
open(fn, "w", encoding="utf-8").write(txt)
|
||||
table_records.append({
|
||||
"index": i, "file": "tables/table-%02d.txt" % i,
|
||||
"sha256": sha256_file(fn), "rows": len(lines),
|
||||
"caption": ex.figcaptions[i - 1] if i - 1 < len(ex.figcaptions) else "",
|
||||
})
|
||||
|
||||
# figures (download)
|
||||
fig_records = []
|
||||
seen = set()
|
||||
n = 0
|
||||
for src in ex.figure_imgs:
|
||||
if src in seen:
|
||||
continue
|
||||
seen.add(src)
|
||||
n += 1
|
||||
if src.startswith("/"):
|
||||
url = "https://ar5iv.labs.arxiv.org" + src
|
||||
elif src.startswith("http"):
|
||||
url = src
|
||||
else:
|
||||
url = "https://ar5iv.labs.arxiv.org/html/" + ARXIV_ID + "/" + src
|
||||
ext = os.path.splitext(src.split("?")[0])[1] or ".png"
|
||||
fn = os.path.join(OUT, "figures", "fig-%02d%s" % (n, ext))
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
||||
data = urllib.request.urlopen(req, timeout=45).read()
|
||||
open(fn, "wb").write(data)
|
||||
ok = True
|
||||
except Exception as e:
|
||||
open(fn, "wb").write(b"")
|
||||
data = b""
|
||||
ok = False
|
||||
print("FIGFETCH FAIL", url, e)
|
||||
fig_records.append({
|
||||
"index": n, "file": "figures/fig-%02d%s" % (n, ext),
|
||||
"url": url, "sha256": sha256_bytes(data), "bytes": len(data),
|
||||
"fetch_ok": ok,
|
||||
"caption": ex.figcaptions[n - 1] if n - 1 < len(ex.figcaptions) else "",
|
||||
})
|
||||
|
||||
# chunking
|
||||
L = len(paper)
|
||||
step = CHUNK - OVERLAP
|
||||
ranges = []
|
||||
i = 0
|
||||
while i < L:
|
||||
ranges.append((i, min(i + CHUNK, L)))
|
||||
i += step
|
||||
chunk_records = []
|
||||
for idx, (s, e) in enumerate(ranges, 1):
|
||||
text = paper[s:e]
|
||||
fn = os.path.join(OUT, "chunks", "chunk-%03d.md" % idx)
|
||||
open(fn, "w", encoding="utf-8").write(text)
|
||||
# needle: a 40-char window unique in paper.txt, preferring digit/uppercase
|
||||
needle = None
|
||||
for off in range(0, max(1, len(text) - 40)):
|
||||
cand = text[off:off + 40]
|
||||
if not cand.strip():
|
||||
continue
|
||||
if paper.count(cand) == 1 and re.search(r"\d", cand):
|
||||
needle = cand
|
||||
break
|
||||
if needle is None:
|
||||
for off in range(0, max(1, len(text) - 40)):
|
||||
cand = text[off:off + 40]
|
||||
if cand.strip() and paper.count(cand) == 1:
|
||||
needle = cand
|
||||
break
|
||||
chunk_records.append({
|
||||
"index": idx, "file": "chunks/chunk-%03d.md" % idx,
|
||||
"start": s, "end": e, "chars": e - s,
|
||||
"sha256": sha256_file(fn), "needle": needle,
|
||||
})
|
||||
|
||||
manifest = {
|
||||
"artifact": "archive-mm-prm-2026-10-22",
|
||||
"arxiv_id": ARXIV_ID,
|
||||
"title": title,
|
||||
"source_url": SRC_URL,
|
||||
"fetched_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"source_sha256": sha256_bytes(src_bytes),
|
||||
"paper_txt_sha256": sha256_file(os.path.join(OUT, "paper.txt")),
|
||||
"paper_chars": L,
|
||||
"chunk_size": CHUNK, "chunk_overlap": OVERLAP,
|
||||
"num_chunks": len(chunk_records),
|
||||
"num_tables": len(table_records),
|
||||
"num_figures": len(fig_records),
|
||||
"headings": ex.headings,
|
||||
"chunks": chunk_records,
|
||||
"tables": table_records,
|
||||
"figures": fig_records,
|
||||
}
|
||||
mp = os.path.join(OUT, "manifest.json")
|
||||
open(mp, "w", encoding="utf-8").write(json.dumps(manifest, indent=2, ensure_ascii=False) + "\n")
|
||||
|
||||
print("title:", title)
|
||||
print("paper_chars:", L, "chunks:", len(chunk_records),
|
||||
"tables:", len(table_records), "figures:", len(fig_records))
|
||||
missing = [c["index"] for c in chunk_records if not c["needle"]]
|
||||
print("chunks_missing_needle:", missing)
|
||||
print("manifest:", mp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,20 @@
|
||||
[2505.13427] MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
|
||||
|
||||
## MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
|
||||
|
||||
Lingxiao Du1 Fanqing Meng1,3∗ Zongkai Liu2∗ Zhixiang Zhou2∗
|
||||
Ping Luo4 Qiaosheng Zhang2,3† Wenqi Shao2,3
|
||||
1Shanghai AI Laboratory 2Shanghai Innovation Institute
|
||||
3Shanghai Jiao Tong University 4The University of Hong Kong
|
||||
|
||||
Thanks: Equal contribution
|
||||
|
||||
Thanks: Corresponding Authors: shaowenqi@pjlab.org.cn; zhangqiaosheng@pjlab.org.cn
|
||||
|
||||
## Abstract
|
||||
|
||||
While Multimodal Large Language Models (MLLMs) have achieved impressive progress in vision-language understanding, they still struggle with complex multi-step reasoning, often producing logically inconsistent or partially correct solutions. A key limitation lies in the lack of fine-grained supervision over intermediate reasoning steps. To address this, we propose MM-PRM, a process reward model trained within a fully automated, scalable framework. We first build MM-Policy, a strong multimodal model trained on diverse mathematical reasoning data. Then, we construct MM-K12, a curated dataset of 10,000 multimodal math problems with verifiable answers, which serves as seed data. Leveraging a Monte Carlo Tree Search (MCTS)-based pipeline, we generate over 700k step-level annotations without human labeling. The resulting PRM is used to score candidate reasoning paths in the Best-of-N inference setup and achieves significant improvements across both in-domain (MM-K12 test set) and out-of-domain (OlympiadBench, MathVista, etc.) benchmarks. Further analysis confirms the effectiveness of soft labels, smaller learning rates, and path diversity in optimizing PRM performance. MM-PRM demonstrates that process supervision is a powerful tool for enhancing the logical robustness of multimodal reasoning systems. We release all our codes and data at https://github.com/ModalMinds/MM-PRM.
|
||||
|
||||
## 1 Introduction
|
||||
|
||||
The rapid advancement of Large Language Models (LLMs) achiam2023gpt; bai2023qwen; team2023gemini; touvron2023llama; touvron2023llama2; grattafiori2024llama; 2023internlm; cai2024internlm2; guo2025deepseek has significantly improved performance on a wide range of natural language processing tasks, including general reasoning and mathematical problem-solving. In parallel, the development of Multimodal Large Language Models (MLLMs) liu2023visual; yao2024minicpm; chen2024expanding; wang2024mpo; gao2024mini; chen2024far; chen2024internvl; Qwen-VL; team2025kimi has unlocked new capabilities in vision-language understanding, showing promising results in areas such as image captioning and visual question answering (VQA). However, despite their impressive capabilities in perception and basic reasoning, MLLMs still struggle with complex multi-step reasoning tasks, particularly in mathematics. These shortcomings often manifest as broken logical chains, inaccurate intermediate steps, or cases that produce incorrect intermediate steps while still
|
||||
@@ -0,0 +1,8 @@
|
||||
reasoning tasks, particularly in mathematics. These shortcomings often manifest as broken logical chains, inaccurate intermediate steps, or cases that produce incorrect intermediate steps while still occasionally arriving at the correct final answer—a phenomenon that introduces high false-positive rates and undermines interpretability.
|
||||
|
||||
To address this issue, reward modeling cobbe2021training; yu2023ovm; uesato2022solving; zang2025internlm has emerged as a promising paradigm. Reward models (RMs) play a central role in reinforcement learning from human feedback (RLHF) ouyang2022training; schulman2017proximal; shao2024deepseekmath; lai2024step; wang2024enhancing; pang2024iterative, and can also be used at inference time to select among multiple candidate responses using Test-Time Scaling (TTS) dong2024rlhf; snell2024scaling; wang2023math; zhang2025lessons; feng2023alphazero; kang2024mindstar; ma2023let; tian2024toward; zhang2024rest strategies such as Best-of-N (BoN).
|
||||
In general, reward models for reasoning tasks can be broadly categorized into two types: Outcome Reward Models (ORMs) and Process Reward Models (PRMs). ORMs cobbe2021training; yu2023ovm provide scalar feedback only on the final answer, overlooking the quality of the intermediate reasoning steps. This limits their ability to guide the model toward robust reasoning paths. In contrast, PRMs li2022making; lightman2023let; uesato2022solving; wang2023math; wang2024multi; luo2024improve offer a more fine-grained approach by evaluating each reasoning step, enabling more accurate and interpretable feedback.
|
||||
|
||||
Some recent works have explored process reward models for mathematical reasoning in pure text. PRM800k lightman2023let manually constructs a large-scale dataset with step-level correctness labels, which is hard to scale. MathShepherd wang2023math adopts Monte Carlo estimation to label reasoning steps by evaluating whether continuations from a given step lead to the correct answer, but its efficiency is relatively low. OmegaPRM luo2024improve introduces a Monte Carlo Tree Search (MCTS)-based framework that enables efficient automated generation of process supervision data. However, all these works focus on mathematical reasoning in pure text. In the field of multimodal mathematical reasoning, how to design an efficient framework for process supervision data generation and stably train process reward models remains a challenging problem.
|
||||
|
||||
To address these challenges, we propose MM-PRM, a powerful process reward model that effectively handles both in-domain and out-of-domain problems. Specifically, we first collect a large-scale, high-quality reasoning dataset and refine its format to train a policy model called MM-Policy with strong reasoning abilities and well-structured output. Inspired by OmegaPRM, we then establish an efficient multimodal process annotation framework. We first collect 10k high-quality K-12-level multimodal mathematical reasoning samples, named MM-K12. Starting with only
|
||||
@@ -0,0 +1,25 @@
|
||||
egaPRM, we then establish an efficient multimodal process annotation framework. We first collect 10k high-quality K-12-level multimodal mathematical reasoning samples, named MM-K12. Starting with only 10k math problems as seed data, our framework automatically generates over 700k step-level annotations without any human supervision. Finally, we train MM-PRM with these data and evaluate its performance on multiple benchmarks using the BoN approach.
|
||||
|
||||
MM-PRM demonstrates strong performance and generalization across multiple benchmarks. Although we only use process data generated from K-12-level math problems for training, MM-PRM achieves remarkable results with MM-Policy on benchmarks such as MathVista that improves from 62.93% to 67.60%. In addition, even though MM-PRM is trained on data produced by MM-Policy, it still delivers competitive results on other models like InternVL2.5-8B and InternVL2.5-78B. For instance, on the self-collected MM-K12 test set, InternVL2.5-8B improves from 27.01% to 37.80%, and on MathVerse, InternVL2.5-78B improves from 50.18% to 54.47%. These results highlight MM-PRM’s ability to generalize both across datasets and across model size, despite being trained on limited and fixed data sources.
|
||||
|
||||
Finally, we provide a detailed discussion on key factors in PRM training, including learning rate and the choice between soft labels and hard labels. We find that the following factors are essential for stable PRM training: (1) using a small learning rate to ensure stable optimization; (2) adopting soft labels instead of hard thresholding to reduce noise and preserve step-wise uncertainty.
|
||||
|
||||
In summary, our contributions are as follows:
|
||||
|
||||
•
|
||||
|
||||
We collect and release MM-K12, a curated multimodal math dataset containing 10,000 seed problems and 500 test problems, all verified to contain unique, checkable answers.
|
||||
|
||||
•
|
||||
|
||||
We build a policy model named MM-Policy using a large corpus of mathematical reasoning data, and develop a fully automated, MCTS-based pipeline for scalable process supervision generation. Leveraging MM-Policy and only 10k seed problems from MM-K12, our framework produces over 700k step-level annotations without human supervision. Based on this, we train MM-PRM, which demonstrates strong performance across multiple benchmarks—for example, boosting accuracy on the MM-K12 test set from 33.92% to 42.80%, on MathVista from 62.93% to 67.60%, and on OlympiadBench from 15.41% to 24.00%.
|
||||
|
||||
•
|
||||
|
||||
We provide a detailed discussion on the settings of key parameters such as the type of process labels and the learning rate. We summarize the key factors for stable PRM training, which can help the community train PRM models more effectively.
|
||||
|
||||
## 2 Related work
|
||||
|
||||
## 2.1 Mathematical reasoning and reward modeling in LLMs
|
||||
|
||||
Mathematical reasoning has become a focal point in evaluating the deep logical abilities of LLMs. cobbe2021training; hendrycks2021measuring. Unlike general language tasks, mathematical problems demand precise multi-step reasoning
|
||||
@@ -0,0 +1,18 @@
|
||||
ome a focal point in evaluating the deep logical abilities of LLMs. cobbe2021training; hendrycks2021measuring. Unlike general language tasks, mathematical problems demand precise multi-step reasoning and logical coherence, motivating approaches like Chain-of-Thought (CoT) promptingwei2022chain and Self-Consistency sampling wang2022self. Supervised fine-tuning (SFT) with structured solution datasets further reinforces reasoning performance, as demonstrated by models like Qwen bai2023qwen, InternLM 2023internlm, and Gemini team2023gemini. More recently, reinforcement learning (RL)-based optimization, exemplified by OpenAI’s o1 jaech2024openai and DeepSeek R1 guo2025deepseek, has emerged as a powerful strategy that surpassing SFT-only baselines.
|
||||
|
||||
However, despite these advances, current LLMs still frequently produce logically inconsistent reasoning steps or false-positive solutions. To address this issue, reward modeling techniques have emerged. Traditional Outcome Reward Models (ORMs)cobbe2021training; yu2023ovm, which assign rewards based solely on the final answer, fail to detect flawed intermediate reasoning. Process Reward Models (PRMs)lightman2023let; wang2023math; wang2024multi; luo2024improve overcome this limitation by explicitly providing step-level supervision, significantly improving logical coherence. Representative works such as PRM800K lightman2023let, MathShepherd wang2023math, and MiPS wang2024multi have demonstrated the effectiveness of PRMs in enhancing mathematical reasoning.
|
||||
|
||||
Overall, PRMs have become an essential approach for improving the robustness and logical consistency of LLM reasoning through fine-grained, step-level evaluation.
|
||||
|
||||
## 2.2 Process supervision data construction
|
||||
|
||||
Training PRMs requires large-scale, high-quality step-level annotations that indicate the correctness of each intermediate reasoning step. However, most publicly available datasets focus solely on final answers, making it challenging to obtain supervision signals for reasoning quality. Existing process supervision construction methods can be broadly categorized into three classes:
|
||||
|
||||
Manual Annotation.
|
||||
The earliest efforts, such as PRM800K lightman2023let, relied on human experts to label each reasoning step. While this yields high-quality annotations, the approach is costly, time-consuming, and difficult to scale—particularly for domains like mathematics where expert knowledge is required.
|
||||
|
||||
Monte Carlo Estimation.
|
||||
Works like MathShepherd wang2023math and MiPS wang2024multi automate supervision by sampling multiple rollouts from each reasoning prefix and estimating step correctness based on the proportion of successful completions. If continuations from a given step often lead to correct answers, that step is considered reliable. This method is simple but suffers from label instability due to high variance in sampled rollouts—especially on long or complex reasoning chains.
|
||||
|
||||
Monte Carlo Tree Search (MCTS).
|
||||
To address these limitations, OmegaPRM luo2024im
|
||||
@@ -0,0 +1,27 @@
|
||||
uffers from label instability due to high variance in sampled rollouts—especially on long or complex reasoning chains.
|
||||
|
||||
Monte Carlo Tree Search (MCTS).
|
||||
To address these limitations, OmegaPRM luo2024improve introduces a structured, search-based alternative. Using a divide-and-conquer MCTS algorithm, it efficiently identifies the first error in a reasoning chain and constructs a tree of reasoning paths with dynamically updated statistics. This approach improves annotation stability, better handles deep reasoning tasks, and generates large volumes of fine-grained supervision from limited seed data without human input.
|
||||
|
||||
Each method reflects a trade-off between cost, scalability, and label fidelity. Among them, MCTS-based approaches have recently emerged as the most effective strategy for generating robust and scalable process supervision, especially for multi-step mathematical reasoning tasks.
|
||||
|
||||
## 3 Method
|
||||
|
||||
## 3.1 Overview
|
||||
|
||||
We present a scalable and fully automated framework for training PRMs tailored to multimodal mathematical reasoning. Our approach addresses two critical bottlenecks in prior work: the lack of step-level supervision and the inefficiency of data generation in complex reasoning tasks. To overcome these challenges, we introduce a structured three-stage pipeline that enables fine-grained reward modeling without any human annotation.
|
||||
As shown in Figure 1, the framework comprises three interconnected stages as follows:
|
||||
|
||||
(1) Policy Model Construction, where a multimodal policy model is trained to generate high-quality reasoning traces following the CoT paradigm.
|
||||
|
||||
(2) Process Supervision Data Generation, where we use a MCTS-based engine, OmegaPRM luo2024improve, to efficiently identify reasoning flaws and produce step-level reward labels at scale.
|
||||
|
||||
(3) Process Reward Model Training, where a PRM is trained to evaluate each reasoning step and provide dense feedback.
|
||||
|
||||
This end-to-end design ensures that process supervision can be generated, modeled, and applied in a fully closed loop. It significantly improves reasoning quality and robustness, especially in tasks requiring long logical chains. In Sections 3.2–3.3 below, we elaborate on these three stages.
|
||||
|
||||
## 3.2 Policy model construction
|
||||
|
||||
The policy model serves as the foundation of our framework, responsible for generating candidate reasoning trajectories given multimodal math problems. These trajectories are later evaluated and labeled to form step-level supervision for training the PRM. Ensuring that the policy model produces logically coherent and structurally complete outputs is thus essential for the effectiveness of the entire system.
|
||||
|
||||
To train the policy model, we curated a large-scale, high-quality dataset of mathematical problems spanning a wide range of topics and difficulty levels. Our dataset integrates samples from over a dozen public math datasets, including R-CoT deng2024r, MAVIS zhang2024mavis, MathV360K shi2024math, NuminaMath li2024numinamath, and DART-Math tong2024dart, w
|
||||
@@ -0,0 +1,13 @@
|
||||
ur dataset integrates samples from over a dozen public math datasets, including R-CoT deng2024r, MAVIS zhang2024mavis, MathV360K shi2024math, NuminaMath li2024numinamath, and DART-Math tong2024dart, with problems ranging from elementary school arithmetic to advanced geometry and statistics. The full list of data sources and sample counts used for training is provided in Appendix A.
|
||||
|
||||
Once collected, all data undergo rigorous cleaning and format standardization. Visual and textual content are paired explicitly, and reasoning traces are reformatted to follow a structured CoT schema, with each logical step clearly marked using structured tags such as <step></step>, and the final conclusion labeled with <answer></answer>. To enhance quality and clarity, we leveraged a strong instruction-tuned language model (i.e., Qwen2.5-72B-Instruct bai2023qwen) to parse original solutions and restructure them into coherent, modular steps. This structured representation not only enhances model learnability but also lays the foundation for generating step-level reward labels in the next stage. The prompt used for solution restructuring is provided in Appendix B.
|
||||
|
||||
With this cleaned and annotated corpus of over 5 million examples, we fine-tuned a strong open-source multimodal model, InternVL2.5-8B chen2024internvl, using supervised learning. This ensures that the model learns to produce logically sound and well-structured outputs that conform to the CoT reasoning pattern.
|
||||
|
||||
The resulting model not only delivers high-quality reasoning trajectories for downstream data annotation but also supports inference-time use cases such as BoN selection with reward-based reranking. Its strong generative ability and logical consistency form the cornerstone of our scalable process supervision framework.
|
||||
|
||||
## 3.3 Process supervision data generation
|
||||
|
||||
To enable fine-grained supervision for step-level reasoning, we adopt an automated process annotation pipeline based on the OmegaPRM luo2024improve framework. OmegaPRM introduces a MCTS-based mechanism for efficiently identifying and labeling intermediate reasoning steps with confidence estimates. Although originally developed for textual mathematical reasoning, we adapt and extend this framework to handle multimodal inputs.
|
||||
|
||||
Our process begins with collecting a curated dataset of 10,000 multimodal math problems from real-world named MM-K12—consisting of 5,000 fill-in-the-blank and 5,000 multiple-choice questions. These problems span a range of curriculum topics from elementary to high school and serve as seed instances for process supervision generation. All examples in MM-K12 are carefully filtered to ensure that each question includes meaningful visual input and has a unique, verifiable answer, making them well-suited for structured reasoning and reward modeling. In addition, MM-K12 also provides an independent test set of 500 problems, constructed under the same criteria, which we use to evaluate in-distribution performance later. For ea
|
||||
@@ -0,0 +1,15 @@
|
||||
and reward modeling. In addition, MM-K12 also provides an independent test set of 500 problems, constructed under the same criteria, which we use to evaluate in-distribution performance later. For each problem, the policy model produces multiple candidate solutions following the CoT paradigm, and these reasoning paths form the raw material for subsequent reward annotation.
|
||||
|
||||
To evaluate the correctness of each intermediate step, we follow OmegaPRM’s hierarchical rollout and search protocol. Specifically, we generate multiple completions (rollouts) from partial prefixes and estimate the correctness of a given step based on whether its downstream completions reach the correct final answer. By applying binary search, the algorithm efficiently pinpoints the earliest step at which the reasoning begins to deviate. These supervision signals are then aggregated within a structured state-action tree, which records the Monte Carlo (MC) estimates and other statistics at each reasoning state. In our implementation, we maintain the full multimodal context—including both textual and visual components—throughout the tree construction and search process.
|
||||
|
||||
Importantly, our adaptation retains the efficiency of OmegaPRM’s divide-and-conquer search while enabling reward supervision for reasoning steps that are conditioned on complex visual stimuli. Through this pipeline, we generate over 700,000 step-level annotations from only 10k seed questions, without requiring manual labeling. The resulting dataset provides dense, high-quality process supervision aligned with real-world multimodal reasoning.
|
||||
|
||||
## 3.4 Process reward model training
|
||||
|
||||
With large-scale step-level supervision in place, we proceed to train a PRM that can assess the quality of reasoning steps given a multimodal context. The PRM is designed to serve as a fine-grained critic, assigning a reward score to each intermediate step conditioned on its preceding reasoning context that enables both test-time scaling and potential RL applications.
|
||||
|
||||
## 3.4.1 Labeling strategy
|
||||
|
||||
A central design decision in PRM training lies in how to formulate supervision signals from the MC estimations luo2024improve; zhang2025lessons. Rather than adopting hard binary label (e.g., y^=1[MC(s)>τ]\hat{y}=1[\mathrm{MC}(s)>\tau]), we use soft label, directly taking the MC scores as continuous supervision targets.
|
||||
|
||||
This choice is motivated by the observation that the MC score reflects more than the correctness of an intermediate step. It also encodes factors such as problem difficulty, step criticality, and distributional uncertainty in the policy model’s rollouts. For instance, a reasoning step within a highly ambiguous or visually complex problem may yield lower MC scores even if the logic is fundamentally sound. In such cases, hard-thresholding may misrepresent the step’s quality, introducing noise into training. By contrast, soft labels preserve the probabilistic nuance and enable smoother learning dynamics, more details will be discu
|
||||
@@ -0,0 +1,28 @@
|
||||
lding may misrepresent the step’s quality, introducing noise into training. By contrast, soft labels preserve the probabilistic nuance and enable smoother learning dynamics, more details will be discussed in section 5.3
|
||||
|
||||
Formally, for each reasoning step xtx_{t} in a path x=[x1,x2,…,xT]x=[x_{1},x_{2},\dots,x_{T}], we assign a supervision target y^t=MC(x<t)∈[0,1]\hat{y}_{t}=\mathrm{MC}(x_{<t})\in[0,1], where MC(x<t)\mathrm{MC}(x_{<t}) denotes the estimated probability that a correct final answer can be reached from this partial path.
|
||||
|
||||
## 3.4.2 Model design and training objective
|
||||
|
||||
To model the prediction task, we treat the PRM as a classifier operating on each reasoning step. Given a multimodal input qq and a generated reasoning trace [x1,x2,…,xT][x_{1},x_{2},\dots,x_{T}], we interleave a special marker token, denoted σ\sigma, after each step, producing an input sequence of the form:
|
||||
|
||||
In our implementation, σ\sigma is instantiated as the token <prm>. At each occurrence of σ\sigma, the model is tasked with producing a scalar confidence score indicating the likelihood that the immediately preceding step is logically correct. Formally, let zYes(i)z^{(i)}_{\texttt{Yes}} and zNo(i)z^{(i)}_{\texttt{No}} denote the unnormalized logits for binary labels “Yes” (correct) and “No” (incorrect) at the ii-th occurrence of σ\sigma. The model’s predicted probability is computed via softmax:
|
||||
|
||||
The training objective is to minimize the cross-entropy loss between the predicted scores p(i)p^{(i)} and the soft labels y^(i)\hat{y}^{(i)}, across all scoring points:
|
||||
|
||||
This formulation guides the model to make fine-grained assessments of reasoning steps, assigning higher confidence to those with stronger evidence of correctness.
|
||||
|
||||
## 4 Experiments
|
||||
|
||||
## 4.1 Experiments setup
|
||||
|
||||
To validate the effectiveness of our proposed process reward modeling framework, we conduct a series of experiments, carefully configured to ensure fair, scalable, and reproducible results.
|
||||
|
||||
Policy model construction.
|
||||
Our policy model(i.e., MM-Policy) is initialized from the multimodal backbone InternVL 2.5-8B and fine-tuned using approximately 4 million cleaned, structured math problems. The model is trained for 1 epoch with a batch size of 128 and a learning rate of 4e-5, updating only the language module while keeping the vision encoder frozen.
|
||||
|
||||
Process supervision data generation.
|
||||
We adapt the OmegaPRMluo2024improve pipeline for multimodal reasoning and apply it to MM-K12 (10k samples). Using MCTS-based structured rollouts, we generate approximately 747,000 step-level annotations. The sampling parameters are tuned for balance between diversity and efficiency: temperature=1.0temperature=1.0, topk=50top_{k}=50, topp=0.9top_{p}=0.9, exploration coefficient cpuct=0.125c_{\text{puct}}=0.125, and up to 200 search steps or 1,000 total rollouts per problem.
|
||||
|
||||
Process reward model training.
|
||||
We initialize the PRM from the fine-tuned policy model and train it for 1 epochs with a batch size of 512 a
|
||||
@@ -0,0 +1,18 @@
|
||||
up to 200 search steps or 1,000 total rollouts per problem.
|
||||
|
||||
Process reward model training.
|
||||
We initialize the PRM from the fine-tuned policy model and train it for 1 epochs with a batch size of 512 and a learning rate of 4e-6.
|
||||
|
||||
## 4.2 Evaluation strategies and benchmarks
|
||||
|
||||
To assess the effectiveness of the proposed MM-PRM in improving reasoning quality, we adopt the BoN evaluation protocol. For each test problem, the policy model generates N=16N=16 candidate reasoning paths independently. The PRM then scores each path step-by-step, producing a sequence of floating-point values representing the predicted quality of each intermediate step, the path with the highest score is selected as the final answer.
|
||||
|
||||
Since PRM outputs a vector of step-wise confidence scores for each candidate path, a crucial component of our evaluation is the aggregation function wang2024multi used to compress this vector into a scalar. We explore a diverse set of aggregation functions, including Min, Average, Max, SumLogPr (i.e., sum of log-probabilities), SumLogOdds (i.e., sum of log-odds), and MeanOdds (i.e., mean odds), each function captures different aspects of path quality. Additionally, a Random baseline is used for comparison, where the final answer is randomly sampled from the same set of 16 candidates. Formal definitions of all aggregation functions are provided in Appendix C.
|
||||
|
||||
We evaluate performance using answer accuracy, defined as the proportion of final selected answers that match the ground truth. This metric directly reflects MM-PRM’s utility in guiding the selection of correct reasoning paths. To comprehensively evaluate our model’s performance and generalization, we conduct experiments on a range of multimodal math benchmarks, including MM-K12 (test set), OlympiadBench (OE_MM_maths_en_COMP) he2024olympiadbench, MathVista(testmini) lu2023mathvista, MathVerse(testmini) zhang2024mathverse, and MathVision(test) wang2024measuring. The MM-K12 test set serves as an in-distribution evaluation. For out-of-distribution assessment, we use the OE_MM_maths_en_COMP split of OlympiadBench, which contains open-ended multimodal questions from international math competitions, closely related in format to MM-K12 but significantly harder. To further test generalization, we include MathVista, which covers a wide range of visual mathematical tasks; MathVerse, which emphasizes understanding structured visual content; and MathVision, which targets abstract visual reasoning. These benchmarks provide a diverse and rigorous setting to measure both performance and generalization of our process reward modeling framework.
|
||||
|
||||
## 4.3 Quantitative results
|
||||
|
||||
We evaluate the effectiveness of MM-PRM by applying it to a range of policy models and testing its impact across multiple multimodal math benchmarks.
|
||||
|
||||
Across all models, MM-PRM yields substantial performance improvements. For example, when applied to MM-Policy on the MM-K12 test set, accuracy improves from 33.92% to 42.80%, and similar gai
|
||||
@@ -0,0 +1,23 @@
|
||||
chmarks.
|
||||
|
||||
Across all models, MM-PRM yields substantial performance improvements. For example, when applied to MM-Policy on the MM-K12 test set, accuracy improves from 33.92% to 42.80%, and similar gains are observed with InternVL2.5-8B, where performance rises from 27.01% to 37.80%. These results confirm that MM-PRM is highly effective at identifying high-quality reasoning paths.
|
||||
|
||||
Beyond in-domain settings, we observe that MM-PRM also generalizes well to larger models and more challenging datasets. As shown in Table 1, applying MM-PRM to InternVL2.5-78B improves accuracy on OlympiadBench from 30.98% to 34.67%, and on MathVerse from 50.18% to 54.47%. Despite being trained only on the small MM-K12 seed dataset and with a fixed policy model(i.e., MM-Policy), MM-PRM consistently enhances reasoning accuracy across diverse benchmarks and models. This demonstrates the potential of scalable step-level reward modeling to improve mathematical reasoning in a model-agnostic and data-efficient manner. Detailed evaluation results across all aggregation functions are provided in Appendix D.
|
||||
|
||||
## 4.4 Qualitative results
|
||||
|
||||
To further illustrate the behavior of MM-PRM, we present a qualitative example involving a geometry problem with parallel lines and an angle bisector. As shown in Figure 2, the policy model generates a four-step reasoning path, which ultimately leads to an incorrect final answer.
|
||||
|
||||
The PRM assigns high scores to the first two steps (0.83 and 0.68), indicating that they are logically sound. In contrast, Step-3 receives a very low score (0.02), signaling that the model has identified a significant reasoning error at this point. This flawed step leads directly to an incorrect conclusion in Step-4.
|
||||
|
||||
This example demonstrates that MM-PRM is capable of detecting localized logical errors within a reasoning chain, such fine-grained judgment is crucial in selecting high-quality reasoning trajectories and filtering out those with subtle but critical flaws.
|
||||
|
||||
## 5 Discussion
|
||||
|
||||
## 5.1 Candidate path’s impact on PRM performance
|
||||
|
||||
Since the PRM operates purely as a selector in the BoN framework, its performance is inherently bounded by the diversity and quality of candidate reasoning paths produced by the policy model. In other words, PRM cannot improve a flawed generation in BoN—it can only choose among the available options. Therefore, the number of reasoning paths generated per problem directly affects its potential to identify a correct and coherent solution.
|
||||
|
||||
To study this effect, we vary the number of generated reasoning paths N∈{2,22,23,24}N\in\{2,2^{2},2^{3},2^{4}\} and measure the corresponding answer accuracy under the aggregation strategy MeanOdds. As shown in Figure 3(a), increasing NN consistently improves MM-PRM’s performance across both test sets.
|
||||
On the MM-K12 test set, accuracy improves from 38.6% at N=2N=2 to 42.8% at N=16N=16, with gains tapering off beyond N=8N=8.
|
||||
In contrast, on OlympiadBench, accuracy increases more steadily—from 18.4% to 24.
|
||||
@@ -0,0 +1,36 @@
|
||||
e MM-K12 test set, accuracy improves from 38.6% at N=2N=2 to 42.8% at N=16N=16, with gains tapering off beyond N=8N=8.
|
||||
In contrast, on OlympiadBench, accuracy increases more steadily—from 18.4% to 24.0%—as NN grows. This suggests that for harder, more diverse tasks, having a larger pool of reasoning paths is critical for PRM to identify valid solutions.
|
||||
|
||||
## 5.2 Learning rate
|
||||
|
||||
As noted in lightman2023let, finetuning a PRM shifts the language models’ objective from generation to discrimination, making learning rate a critical factor. Smaller learning rates are often preferred to maintain stability and preserve pretrained knowledge.
|
||||
|
||||
We evaluate MM-PRM trained under different learning rates on the MM-K12 test set using MeanOdds aggregator. As shown in Figure 3(b), performance peaks at 4e-6—about one-tenth the learning rate typically used in supervised fine-tuning—then drops sharply at higher values. This confirms that a moderate, conservative learning rate leads to better training, while overly large values degrade accuracy.
|
||||
|
||||
## 5.3 Soft label vs. Hard label
|
||||
|
||||
As discussed in section 3.4.1, we adopt soft label—i.e., real-valued MC scores—as supervision for step-level reward modeling. Unlike hard label, soft label retain uncertainty and allow the model to learn more nuanced representations of reasoning quality.
|
||||
|
||||
To assess this design choice, we compare soft label with hard label thresholding, where steps with MC>0MC>0 are treated as correct, and others as incorrect, following the protocol in luo2024improve; wang2024openr.
|
||||
|
||||
As shown in Table 2, soft-label training consistently outperforms hard-label training across all aggregation strategies. For instance, under the Average aggregator, soft labels yield 43% accuracy on MM-K12 test set, compared to 34.4% with hard labels. Similar improvements are observed with SumLogOdds (43.2% vs. 33.8%) and MeanOdds (42.8% vs. 37.0%).
|
||||
|
||||
## 6 Conclusion
|
||||
|
||||
In this work, we introduce MM-PRM, a process reward model built upon a scalable framework for multimodal mathematical process reward modeling that enables step-level supervision without human annotation. By leveraging a multimodal policy model and an MCTS-based data generation pipeline, we construct over 700k process-level labels from only 10k K-12 math problems in our MM-K12 dataset. Our trained PRM significantly improves reasoning accuracy in BoN inference across both standard and challenging benchmarks, and demonstrates strong generalization to new datasets and models. Extensive analysis further confirms the importance of soft labeling, conservative learning rates, and sufficient path diversity. MM-PRM highlights the value of process supervision for enhancing multimodal mathematical problem-solving.
|
||||
|
||||
## References
|
||||
|
||||
[1]
|
||||
|
||||
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya,
|
||||
Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman,
|
||||
Shyamal Anadkat, et al.
|
||||
|
||||
Gpt-4 technical report.
|
||||
|
||||
arXiv preprint arXiv:2303.08774, 2023.
|
||||
|
||||
[2]
|
||||
|
||||
Jinze Bai, Shu
|
||||
@@ -0,0 +1,126 @@
|
||||
Ahmad, Ilge Akkaya,
|
||||
Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman,
|
||||
Shyamal Anadkat, et al.
|
||||
|
||||
Gpt-4 technical report.
|
||||
|
||||
arXiv preprint arXiv:2303.08774, 2023.
|
||||
|
||||
[2]
|
||||
|
||||
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan,
|
||||
Wenbin Ge, Yu Han, Fei Huang, et al.
|
||||
|
||||
Qwen technical report.
|
||||
|
||||
arXiv preprint arXiv:2309.16609, 2023.
|
||||
|
||||
[3]
|
||||
|
||||
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu,
|
||||
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican,
|
||||
et al.
|
||||
|
||||
Gemini: a family of highly capable multimodal models.
|
||||
|
||||
arXiv preprint arXiv:2312.11805, 2023.
|
||||
|
||||
[4]
|
||||
|
||||
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
|
||||
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
|
||||
Hambro, Faisal Azhar, et al.
|
||||
|
||||
Llama: Open and efficient foundation language models.
|
||||
|
||||
arXiv preprint arXiv:2302.13971, 2023.
|
||||
|
||||
[5]
|
||||
|
||||
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
|
||||
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
|
||||
et al.
|
||||
|
||||
Llama 2: Open foundation and fine-tuned chat models.
|
||||
|
||||
arXiv preprint arXiv:2307.09288, 2023.
|
||||
|
||||
[6]
|
||||
|
||||
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek
|
||||
Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex
|
||||
Vaughan, et al.
|
||||
|
||||
The llama 3 herd of models.
|
||||
|
||||
arXiv preprint arXiv:2407.21783, 2024.
|
||||
|
||||
[7]
|
||||
|
||||
InternLM Team.
|
||||
|
||||
Internlm: A multilingual language model with progressively enhanced
|
||||
capabilities.
|
||||
|
||||
https://github.com/InternLM/InternLM-techreport, 2023.
|
||||
|
||||
[8]
|
||||
|
||||
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen,
|
||||
Zehui Chen, Zhi Chen, Pei Chu, et al.
|
||||
|
||||
Internlm2 technical report.
|
||||
|
||||
arXiv preprint arXiv:2403.17297, 2024.
|
||||
|
||||
[9]
|
||||
|
||||
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu,
|
||||
Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
|
||||
|
||||
Deepseek-r1: Incentivizing reasoning capability in llms via
|
||||
reinforcement learning.
|
||||
|
||||
arXiv preprint arXiv:2501.12948, 2025.
|
||||
|
||||
[10]
|
||||
|
||||
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
|
||||
|
||||
Visual instruction tuning.
|
||||
|
||||
Advances in neural information processing systems,
|
||||
36:34892–34916, 2023.
|
||||
|
||||
[11]
|
||||
|
||||
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi
|
||||
Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al.
|
||||
|
||||
Minicpm-v: A gpt-4v level mllm on your phone.
|
||||
|
||||
arXiv preprint arXiv:2408.01800, 2024.
|
||||
|
||||
[12]
|
||||
|
||||
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo
|
||||
Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
|
||||
|
||||
Expanding performance boundaries of open-source multimodal models
|
||||
with model, data, and test-time scaling.
|
||||
|
||||
arXiv preprint arXiv:2412.05271, 2024.
|
||||
|
||||
[13]
|
||||
|
||||
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo
|
||||
Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai.
|
||||
|
||||
Enhancing the reasoning ability of multimodal large language models
|
||||
via mixed preference optimization.
|
||||
|
||||
arXiv preprint arXiv:2411.10442, 2024.
|
||||
|
||||
[14]
|
||||
|
||||
Zhangwei Gao, Zhe Chen, Erfei Cui, Yi
|
||||
@@ -0,0 +1,116 @@
|
||||
Jifeng Dai.
|
||||
|
||||
Enhancing the reasoning ability of multimodal large language models
|
||||
via mixed preference optimization.
|
||||
|
||||
arXiv preprint arXiv:2411.10442, 2024.
|
||||
|
||||
[14]
|
||||
|
||||
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao
|
||||
Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al.
|
||||
|
||||
Mini-internvl: a flexible-transfer pocket multi-modal model with 5%
|
||||
parameters and 90% performance.
|
||||
|
||||
Visual Intelligence, 2(1):1–17, 2024.
|
||||
|
||||
[15]
|
||||
|
||||
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen
|
||||
Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al.
|
||||
|
||||
How far are we to gpt-4v? closing the gap to commercial multimodal
|
||||
models with open-source suites.
|
||||
|
||||
Science China Information Sciences, 67(12):220101, 2024.
|
||||
|
||||
[16]
|
||||
|
||||
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong,
|
||||
Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.
|
||||
|
||||
Internvl: Scaling up vision foundation models and aligning for
|
||||
generic visual-linguistic tasks.
|
||||
|
||||
In Proceedings of the IEEE/CVF Conference on Computer Vision and
|
||||
Pattern Recognition, pages 24185–24198, 2024.
|
||||
|
||||
[17]
|
||||
|
||||
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang
|
||||
Lin, Chang Zhou, and Jingren Zhou.
|
||||
|
||||
Qwen-vl: A versatile vision-language model for understanding,
|
||||
localization, text reading, and beyond.
|
||||
|
||||
arXiv preprint arXiv:2308.12966, 2023.
|
||||
|
||||
[18]
|
||||
|
||||
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng
|
||||
Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al.
|
||||
|
||||
Kimi k1. 5: Scaling reinforcement learning with llms.
|
||||
|
||||
arXiv preprint arXiv:2501.12599, 2025.
|
||||
|
||||
[19]
|
||||
|
||||
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
|
||||
Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
|
||||
et al.
|
||||
|
||||
Training verifiers to solve math word problems.
|
||||
|
||||
arXiv preprint arXiv:2110.14168, 2021.
|
||||
|
||||
[20]
|
||||
|
||||
Fei Yu, Anningzhe Gao, and Benyou Wang.
|
||||
|
||||
Ovm, outcome-supervised value models for planning in mathematical
|
||||
reasoning.
|
||||
|
||||
arXiv preprint arXiv:2311.09724, 2023.
|
||||
|
||||
[21]
|
||||
|
||||
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa
|
||||
Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.
|
||||
|
||||
Solving math word problems with process-and outcome-based feedback.
|
||||
|
||||
arXiv preprint arXiv:2211.14275, 2022.
|
||||
|
||||
[22]
|
||||
|
||||
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding,
|
||||
Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al.
|
||||
|
||||
Internlm-xcomposer2. 5-reward: A simple yet effective multi-modal
|
||||
reward model.
|
||||
|
||||
arXiv preprint arXiv:2501.12368, 2025.
|
||||
|
||||
[23]
|
||||
|
||||
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
|
||||
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
|
||||
|
||||
Training language models to follow instructions with human feedback.
|
||||
|
||||
Advances in neural information processing systems,
|
||||
35:27730–27744, 2022.
|
||||
|
||||
[24]
|
||||
|
||||
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
|
||||
|
||||
Proximal policy optimization algorithms.
|
||||
|
||||
arXiv preprint arXiv:1707.06347, 2017.
|
||||
|
||||
[25]
|
||||
|
||||
Zhihong Shao, Peiyi Wang, Qihao
|
||||
@@ -0,0 +1,127 @@
|
||||
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
|
||||
|
||||
Proximal policy optimization algorithms.
|
||||
|
||||
arXiv preprint arXiv:1707.06347, 2017.
|
||||
|
||||
[25]
|
||||
|
||||
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei
|
||||
Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
|
||||
|
||||
Deepseekmath: Pushing the limits of mathematical reasoning in open
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2402.03300, 2024.
|
||||
|
||||
[26]
|
||||
|
||||
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia.
|
||||
|
||||
Step-dpo: Step-wise preference optimization for long-chain reasoning
|
||||
of llms.
|
||||
|
||||
arXiv preprint arXiv:2406.18629, 2024.
|
||||
|
||||
[27]
|
||||
|
||||
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo
|
||||
Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al.
|
||||
|
||||
Enhancing the reasoning ability of multimodal large language models
|
||||
via mixed preference optimization.
|
||||
|
||||
arXiv preprint arXiv:2411.10442, 2024.
|
||||
|
||||
[28]
|
||||
|
||||
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar,
|
||||
and Jason Weston.
|
||||
|
||||
Iterative reasoning preference optimization.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:116617–116637, 2024.
|
||||
|
||||
[29]
|
||||
|
||||
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan
|
||||
Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang.
|
||||
|
||||
Rlhf workflow: From reward modeling to online rlhf.
|
||||
|
||||
arXiv preprint arXiv:2405.07863, 2024.
|
||||
|
||||
[30]
|
||||
|
||||
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
|
||||
|
||||
Scaling llm test-time compute optimally can be more effective than
|
||||
scaling model parameters.
|
||||
|
||||
arXiv preprint arXiv:2408.03314, 2024.
|
||||
|
||||
[31]
|
||||
|
||||
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu,
|
||||
and Zhifang Sui.
|
||||
|
||||
Math-shepherd: Verify and reinforce llms step-by-step without human
|
||||
annotations.
|
||||
|
||||
arXiv preprint arXiv:2312.08935, 2023.
|
||||
|
||||
[32]
|
||||
|
||||
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu,
|
||||
Dayiheng Liu, Jingren Zhou, and Junyang Lin.
|
||||
|
||||
The lessons of developing process reward models in mathematical
|
||||
reasoning.
|
||||
|
||||
arXiv preprint arXiv:2501.07301, 2025.
|
||||
|
||||
[33]
|
||||
|
||||
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan
|
||||
Zhang, and Jun Wang.
|
||||
|
||||
Alphazero-like tree-search can guide large language model decoding
|
||||
and training.
|
||||
|
||||
arXiv preprint arXiv:2309.17179, 2023.
|
||||
|
||||
[34]
|
||||
|
||||
Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong
|
||||
Li, Xu He, Quan He, Feng Wen, et al.
|
||||
|
||||
Mindstar: Enhancing math reasoning in pre-trained llms at inference
|
||||
time.
|
||||
|
||||
arXiv preprint arXiv:2405.16265, 2024.
|
||||
|
||||
[35]
|
||||
|
||||
Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and
|
||||
Hongxia Yang.
|
||||
|
||||
Let’s reward step by step: Step-level reward model as the navigators
|
||||
for reasoning.
|
||||
|
||||
arXiv preprint arXiv:2310.10080, 2023.
|
||||
|
||||
[36]
|
||||
|
||||
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi,
|
||||
and Dong Yu.
|
||||
|
||||
Toward self-improvement of llms via imagination, searching, and
|
||||
criticizing.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:52723–52748, 2024.
|
||||
|
||||
[37]
|
||||
|
||||
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yis
|
||||
@@ -0,0 +1,121 @@
|
||||
|
||||
Toward self-improvement of llms via imagination, searching, and
|
||||
criticizing.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:52723–52748, 2024.
|
||||
|
||||
[37]
|
||||
|
||||
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang.
|
||||
|
||||
Rest-mcts*: Llm self-training via process reward guided tree search.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:64735–64772, 2024.
|
||||
|
||||
[38]
|
||||
|
||||
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and
|
||||
Weizhu Chen.
|
||||
|
||||
Making large language models better reasoners with step-aware
|
||||
verifier.
|
||||
|
||||
arXiv preprint arXiv:2206.02336, 2022.
|
||||
|
||||
[39]
|
||||
|
||||
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker,
|
||||
Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
|
||||
|
||||
Let’s verify step by step.
|
||||
|
||||
In The Twelfth International Conference on Learning
|
||||
Representations, 2023.
|
||||
|
||||
[40]
|
||||
|
||||
Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and
|
||||
Jingbo Shang.
|
||||
|
||||
Multi-step problem solving through a verifier: An empirical analysis
|
||||
on model-induced process supervision.
|
||||
|
||||
arXiv preprint arXiv:2402.02658, 2024.
|
||||
|
||||
[41]
|
||||
|
||||
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara,
|
||||
Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al.
|
||||
|
||||
Improve mathematical reasoning in language models by automated
|
||||
process supervision.
|
||||
|
||||
arXiv preprint arXiv:2406.06592, 2024.
|
||||
|
||||
[42]
|
||||
|
||||
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
|
||||
Tang, Dawn Song, and Jacob Steinhardt.
|
||||
|
||||
Measuring mathematical problem solving with the math dataset.
|
||||
|
||||
arXiv preprint arXiv:2103.03874, 2021.
|
||||
|
||||
[43]
|
||||
|
||||
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
|
||||
Le, Denny Zhou, et al.
|
||||
|
||||
Chain-of-thought prompting elicits reasoning in large language
|
||||
models.
|
||||
|
||||
Advances in neural information processing systems,
|
||||
35:24824–24837, 2022.
|
||||
|
||||
[44]
|
||||
|
||||
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang,
|
||||
Aakanksha Chowdhery, and Denny Zhou.
|
||||
|
||||
Self-consistency improves chain of thought reasoning in language
|
||||
models.
|
||||
|
||||
arXiv preprint arXiv:2203.11171, 2022.
|
||||
|
||||
[45]
|
||||
|
||||
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden
|
||||
Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
|
||||
|
||||
Openai o1 system card.
|
||||
|
||||
arXiv preprint arXiv:2412.16720, 2024.
|
||||
|
||||
[46]
|
||||
|
||||
Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang,
|
||||
Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, et al.
|
||||
|
||||
R-cot: Reverse chain-of-thought problem generation for geometric
|
||||
reasoning in large multimodal models.
|
||||
|
||||
arXiv preprint arXiv:2410.17885, 2024.
|
||||
|
||||
[47]
|
||||
|
||||
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang,
|
||||
Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al.
|
||||
|
||||
Mavis: Mathematical visual instruction tuning with an automatic data
|
||||
engine.
|
||||
|
||||
arXiv preprint arXiv:2407.08739, 2024.
|
||||
|
||||
[48]
|
||||
|
||||
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong
|
||||
Bing, and Roy Ka-Wei Lee.
|
||||
|
||||
Math-llava: Bootstrapping mathematical reasoning for multimodal l
|
||||
@@ -0,0 +1,118 @@
|
||||
arXiv:2407.08739, 2024.
|
||||
|
||||
[48]
|
||||
|
||||
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong
|
||||
Bing, and Roy Ka-Wei Lee.
|
||||
|
||||
Math-llava: Bootstrapping mathematical reasoning for multimodal large
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2406.17294, 2024.
|
||||
|
||||
[49]
|
||||
|
||||
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi
|
||||
Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al.
|
||||
|
||||
Numinamath: The largest public dataset in ai4maths with 860k pairs of
|
||||
competition math problems and solutions.
|
||||
|
||||
Hugging Face repository, 13:9, 2024.
|
||||
|
||||
[50]
|
||||
|
||||
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He.
|
||||
|
||||
Dart-math: Difficulty-aware rejection tuning for mathematical
|
||||
problem-solving.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:7821–7846, 2024.
|
||||
|
||||
[51]
|
||||
|
||||
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen,
|
||||
Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
|
||||
|
||||
Olympiadbench: A challenging benchmark for promoting agi with
|
||||
olympiad-level bilingual multimodal scientific problems.
|
||||
|
||||
arXiv preprint arXiv:2402.14008, 2024.
|
||||
|
||||
[52]
|
||||
|
||||
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh
|
||||
Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
|
||||
|
||||
Mathvista: Evaluating mathematical reasoning of foundation models in
|
||||
visual contexts.
|
||||
|
||||
arXiv preprint arXiv:2310.02255, 2023.
|
||||
|
||||
[53]
|
||||
|
||||
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu,
|
||||
Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al.
|
||||
|
||||
Mathverse: Does your multi-modal llm truly see the diagrams in visual
|
||||
math problems?
|
||||
|
||||
In European Conference on Computer Vision, pages 169–186.
|
||||
Springer, 2024.
|
||||
|
||||
[54]
|
||||
|
||||
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie
|
||||
Zhan, and Hongsheng Li.
|
||||
|
||||
Measuring multimodal mathematical reasoning with math-vision dataset.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:95095–95169, 2024.
|
||||
|
||||
[55]
|
||||
|
||||
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong,
|
||||
Yan Song, Lei Chen, Lionel M Ni, et al.
|
||||
|
||||
Openr: An open source framework for advanced reasoning with large
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2410.09671, 2024.
|
||||
|
||||
[56]
|
||||
|
||||
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang.
|
||||
|
||||
Multimath: Bridging visual and mathematical reasoning for large
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2409.00147, 2024.
|
||||
|
||||
[57]
|
||||
|
||||
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and
|
||||
Qi Liu.
|
||||
|
||||
Multimodal arxiv: A dataset for improving scientific comprehension of
|
||||
large vision-language models.
|
||||
|
||||
arXiv preprint arXiv:2403.00231, 2024.
|
||||
|
||||
[58]
|
||||
|
||||
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng.
|
||||
|
||||
Math-puma: Progressive upward multimodal alignment to enhance
|
||||
mathematical reasoning.
|
||||
|
||||
In Proceedings of the AAAI Conference on Artificial
|
||||
Intelligence, volume 39, pages 26183–26191, 2025.
|
||||
|
||||
[59]
|
||||
|
||||
Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan,
|
||||
Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, et al.
|
||||
|
||||
Multimodal
|
||||
@@ -0,0 +1,107 @@
|
||||
telligence, volume 39, pages 26183–26191, 2025.
|
||||
|
||||
[59]
|
||||
|
||||
Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan,
|
||||
Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, et al.
|
||||
|
||||
Multimodal self-instruct: Synthetic abstract image and visual
|
||||
reasoning instruction using language model.
|
||||
|
||||
arXiv preprint arXiv:2407.07053, 2024.
|
||||
|
||||
[60]
|
||||
|
||||
Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li,
|
||||
Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang,
|
||||
Yu Qiao, and Jifeng Dai.
|
||||
|
||||
Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024.
|
||||
|
||||
[61]
|
||||
|
||||
Cyrile Delestre Tom Agonnoude, 2024.
|
||||
|
||||
[62]
|
||||
|
||||
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and
|
||||
Hannaneh Hajishirzi.
|
||||
|
||||
Mathqa: Towards interpretable math word problem solving with
|
||||
operation-based formalisms.
|
||||
|
||||
arXiv preprint arXiv:1905.13319, 2019.
|
||||
|
||||
[63]
|
||||
|
||||
PawanKrd.
|
||||
|
||||
math-gpt-4o-200k.
|
||||
|
||||
Hugging Face repository, 2024.
|
||||
|
||||
[64]
|
||||
|
||||
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and
|
||||
Wenhu Chen.
|
||||
|
||||
Mammoth: Building math generalist models through hybrid instruction
|
||||
tuning.
|
||||
|
||||
arXiv preprint arXiv:2309.05653, 2023.
|
||||
|
||||
## Appendix
|
||||
|
||||
## Appendix A Policy Model training data
|
||||
|
||||
Table 3 summarizes the datasets and sample counts used to train the policy model.
|
||||
|
||||
## Appendix B Prompt template for data cleaning
|
||||
|
||||
The prompt template used for data cleaning is shown below. Before generating logical steps and the final answer, the language model is first instructed to explicitly identify the key knowledge points involved in the problem, along with brief explanations. This two-stage prompting strategy enhances the model’s understanding of the question prior to solution restructuring.
|
||||
|
||||
## Appendix C Aggregation function definitions
|
||||
|
||||
In the BoN inference setup, each reasoning path is assigned a vector of step-level scores predicted by the PRM. To compare and rank these paths, we apply an aggregation function to compress each score vector into a single scalar value, following the design considerations discussed in wang2024multi. The following aggregation strategies are used in our experiments:
|
||||
|
||||
•
|
||||
|
||||
Min: Computes the minimum of all step scores:
|
||||
|
||||
•
|
||||
|
||||
Max: Computes the maximum of all step scores:
|
||||
|
||||
•
|
||||
|
||||
Average: Computes the arithmetic mean of all step scores:
|
||||
|
||||
•
|
||||
|
||||
SumLogPr: Computes the sum of log-probabilities of all step scores:
|
||||
|
||||
•
|
||||
|
||||
SumLogOdds: Computes the sum of log-odds of all step scores:
|
||||
|
||||
•
|
||||
|
||||
MeanOdds: Computes the mean of odds-transformed step scores:
|
||||
|
||||
Here, rjr_{j} denotes the jj-th reasoning path consisting of TjT_{j} steps, and pi(j)∈(0,1)p_{i}^{(j)}\in(0,1) is the predicted correctness probability of the ii-th step in that path.
|
||||
|
||||
## Appendix D Full evaluation results
|
||||
|
||||
## D.1 MM-Policy
|
||||
|
||||
## D.2 InternVL-8B
|
||||
|
||||
## D.3 InternVL-26B
|
||||
|
||||
## D.4 InternVL-38B
|
||||
|
||||
## D.5 InternVL-78B
|
||||
|
||||
## Appendix E Limitations
|
||||
|
||||
While our proposed framework demonstrates strong performance and generalization across multiple benchmarks, it also has several limitations: (1) Due to com
|
||||
@@ -0,0 +1,17 @@
|
||||
D.5 InternVL-78B
|
||||
|
||||
## Appendix E Limitations
|
||||
|
||||
While our proposed framework demonstrates strong performance and generalization across multiple benchmarks, it also has several limitations: (1) Due to computational constraints, we conduct training only on the InternVL series with 8B parameters, without exploring larger models or architectures from other model families. This restricts our ability to fully assess how PRM training behavior scales with model size or generalizes across different backbones. (2) The seed data used for process supervision generation is limited in diversity, as it consists solely of K-12 math problems. As a result, the PRM may be less exposed to advanced mathematical domains or visual formats beyond the scope of standard educational settings. We leave broader model coverage and more diverse seed data construction as promising directions for future work.
|
||||
|
||||
◄
|
||||
|
||||
Feeling
|
||||
lucky?
|
||||
|
||||
Conversion
|
||||
report
|
||||
Report
|
||||
an issue
|
||||
View original
|
||||
on arXiv►
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 176 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 587 KiB |
@@ -0,0 +1,366 @@
|
||||
{
|
||||
"artifact": "archive-mm-prm-2026-10-22",
|
||||
"arxiv_id": "2505.13427",
|
||||
"title": "MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision",
|
||||
"source_url": "https://ar5iv.labs.arxiv.org/html/2505.13427",
|
||||
"fetched_utc": "2026-10-22T19:57:42Z",
|
||||
"source_sha256": "8eee97ff5bb6331bc4051aea391708324dac576ee7f8d3548512cf1d06fd2b9c",
|
||||
"paper_txt_sha256": "07254c3ac7de910d4d6b1188fd5eed0d57892f608e0778b81533d7eacb6903cf",
|
||||
"paper_chars": 48565,
|
||||
"chunk_size": 3000,
|
||||
"chunk_overlap": 200,
|
||||
"num_chunks": 18,
|
||||
"num_tables": 17,
|
||||
"num_figures": 3,
|
||||
"headings": [
|
||||
"MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision",
|
||||
"Abstract",
|
||||
"1 Introduction",
|
||||
"2 Related work",
|
||||
"2.1 Mathematical reasoning and reward modeling in LLMs",
|
||||
"2.2 Process supervision data construction",
|
||||
"3 Method",
|
||||
"3.1 Overview",
|
||||
"3.2 Policy model construction",
|
||||
"3.3 Process supervision data generation",
|
||||
"3.4 Process reward model training",
|
||||
"3.4.1 Labeling strategy",
|
||||
"3.4.2 Model design and training objective",
|
||||
"4 Experiments",
|
||||
"4.1 Experiments setup",
|
||||
"4.2 Evaluation strategies and benchmarks",
|
||||
"4.3 Quantitative results",
|
||||
"4.4 Qualitative results",
|
||||
"5 Discussion",
|
||||
"5.1 Candidate path’s impact on PRM performance",
|
||||
"5.2 Learning rate",
|
||||
"5.3 Soft label vs. Hard label",
|
||||
"6 Conclusion",
|
||||
"References",
|
||||
"Appendix",
|
||||
"Appendix A Policy Model training data",
|
||||
"Appendix B Prompt template for data cleaning",
|
||||
"Appendix C Aggregation function definitions",
|
||||
"Appendix D Full evaluation results",
|
||||
"D.1 MM-Policy",
|
||||
"D.2 InternVL-8B",
|
||||
"D.3 InternVL-26B",
|
||||
"D.4 InternVL-38B",
|
||||
"D.5 InternVL-78B",
|
||||
"Appendix E Limitations"
|
||||
],
|
||||
"chunks": [
|
||||
{
|
||||
"index": 1,
|
||||
"file": "chunks/chunk-001.md",
|
||||
"start": 0,
|
||||
"end": 3000,
|
||||
"chars": 3000,
|
||||
"sha256": "ff2c2f649c50cec6821f113bc9cc7a32d655e0998ab3ca41c9f5a7e2b0483927",
|
||||
"needle": "[2505.13427] MM-PRM: Enhancing Multimoda"
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"file": "chunks/chunk-002.md",
|
||||
"start": 2800,
|
||||
"end": 5800,
|
||||
"chars": 3000,
|
||||
"sha256": "57e7149803a4179fe018461bf75212acb97106f480e3c018e1c2664da39e05ef",
|
||||
"needle": "dress this issue, reward modeling cobbe2"
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"file": "chunks/chunk-003.md",
|
||||
"start": 5600,
|
||||
"end": 8600,
|
||||
"chars": 3000,
|
||||
"sha256": "03899de35616b8c048ba08ffe625db278b69c2d33270c79968e7bdf1d6e02888",
|
||||
"needle": "annotation framework. We first collect 1"
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"file": "chunks/chunk-004.md",
|
||||
"start": 8400,
|
||||
"end": 11400,
|
||||
"chars": 3000,
|
||||
"sha256": "21c273e21c12ab3c9be3a52d60956ad0a920d468046caec8c05fa38a6f4f4a68",
|
||||
"needle": "e deep logical abilities of LLMs. cobbe2"
|
||||
},
|
||||
{
|
||||
"index": 5,
|
||||
"file": "chunks/chunk-005.md",
|
||||
"start": 11200,
|
||||
"end": 14200,
|
||||
"chars": 3000,
|
||||
"sha256": "e5aada7975b07690a7ebad1b211f7a75f8ee022fa827e6f5de39e066fe97557c",
|
||||
"needle": "address these limitations, OmegaPRM luo2"
|
||||
},
|
||||
{
|
||||
"index": 6,
|
||||
"file": "chunks/chunk-006.md",
|
||||
"start": 14000,
|
||||
"end": 17000,
|
||||
"chars": 3000,
|
||||
"sha256": "5c0c9e0a8a63497a4e48beb2d88fcd71f0d0088cf6dffe6f7658e2d6d2bc6a8b",
|
||||
"needle": "lic math datasets, including R-CoT deng2"
|
||||
},
|
||||
{
|
||||
"index": 7,
|
||||
"file": "chunks/chunk-007.md",
|
||||
"start": 16800,
|
||||
"end": 19800,
|
||||
"chars": 3000,
|
||||
"sha256": "19bfbed45e90a6068395daf7fa13c25afa9bf7790f5828af95e99b567721144b",
|
||||
"needle": " and reward modeling. In addition, MM-K1"
|
||||
},
|
||||
{
|
||||
"index": 8,
|
||||
"file": "chunks/chunk-008.md",
|
||||
"start": 19600,
|
||||
"end": 22600,
|
||||
"chars": 3000,
|
||||
"sha256": "41042830a268973ab3aab3f820e012857b7e5f6102a75edea6a16dd58e3d9bd4",
|
||||
"needle": "e details will be discussed in section 5"
|
||||
},
|
||||
{
|
||||
"index": 9,
|
||||
"file": "chunks/chunk-009.md",
|
||||
"start": 22400,
|
||||
"end": 25400,
|
||||
"chars": 3000,
|
||||
"sha256": "18524098b35d07a0e9d0287c04673f4a18614d060c8e338794906dc8d18bee09",
|
||||
"needle": " up to 200 search steps or 1,000 total r"
|
||||
},
|
||||
{
|
||||
"index": 10,
|
||||
"file": "chunks/chunk-010.md",
|
||||
"start": 25200,
|
||||
"end": 28200,
|
||||
"chars": 3000,
|
||||
"sha256": "2ae8a65333c125d1bb9a11525dcfadd49db11842893d7b35768d004aa944c767",
|
||||
"needle": ", when applied to MM-Policy on the MM-K1"
|
||||
},
|
||||
{
|
||||
"index": 11,
|
||||
"file": "chunks/chunk-011.md",
|
||||
"start": 28000,
|
||||
"end": 31000,
|
||||
"chars": 3000,
|
||||
"sha256": "160c0137b41f3e6ea8609a37028dbf3a29ee3f21af1e331907275a9a4b3d01d3",
|
||||
"needle": "-K12 test set, accuracy improves from 38"
|
||||
},
|
||||
{
|
||||
"index": 12,
|
||||
"file": "chunks/chunk-012.md",
|
||||
"start": 30800,
|
||||
"end": 33800,
|
||||
"chars": 3000,
|
||||
"sha256": "36ea14668909ded3df34eab11a80bde77bac45f2be7f178450d562995461a926",
|
||||
"needle": "m Altman,\nShyamal Anadkat, et al.\n\nGpt-4"
|
||||
},
|
||||
{
|
||||
"index": 13,
|
||||
"file": "chunks/chunk-013.md",
|
||||
"start": 33600,
|
||||
"end": 36600,
|
||||
"chars": 3000,
|
||||
"sha256": "793d9287437e522563b210f9d586392ce3676261321a29c9f9988411517f5a32",
|
||||
"needle": "Xiv preprint arXiv:2411.10442, 2024.\n\n[1"
|
||||
},
|
||||
{
|
||||
"index": 14,
|
||||
"file": "chunks/chunk-014.md",
|
||||
"start": 36400,
|
||||
"end": 39400,
|
||||
"chars": 3000,
|
||||
"sha256": "8b9e4a9280439200ab2b9f1cec2707f805e423602ede0c98dfc26b54438eceb6",
|
||||
"needle": "tion algorithms.\n\narXiv preprint arXiv:1"
|
||||
},
|
||||
{
|
||||
"index": 15,
|
||||
"file": "chunks/chunk-015.md",
|
||||
"start": 39200,
|
||||
"end": 42200,
|
||||
"chars": 3000,
|
||||
"sha256": "e9b9413a1afa9a8e8e2de933b29bc984ec9df00aa68db3d040bfeb67674bc699",
|
||||
"needle": "ral Information Processing Systems,\n37:5"
|
||||
},
|
||||
{
|
||||
"index": 16,
|
||||
"file": "chunks/chunk-016.md",
|
||||
"start": 42000,
|
||||
"end": 45000,
|
||||
"chars": 3000,
|
||||
"sha256": "2b57cc8a79e3cc54d5ca40e5ff655f7b73c2387cd4ac1a3f2807fd23857c50a3",
|
||||
"needle": "arXiv:2407.08739, 2024.\n\n[48]\n\nWenhao Sh"
|
||||
},
|
||||
{
|
||||
"index": 17,
|
||||
"file": "chunks/chunk-017.md",
|
||||
"start": 44800,
|
||||
"end": 47800,
|
||||
"chars": 3000,
|
||||
"sha256": "8e9ccb1a66ab35e83ef0fbd7e17efe326339924c80ab00195efc71992c065a5e",
|
||||
"needle": "telligence, volume 39, pages 26183–26191"
|
||||
},
|
||||
{
|
||||
"index": 18,
|
||||
"file": "chunks/chunk-018.md",
|
||||
"start": 47600,
|
||||
"end": 48565,
|
||||
"chars": 965,
|
||||
"sha256": "b93fce7702332a3664c70b82d4257fe4b911fef4824685ea40a213607552eab0",
|
||||
"needle": "D.5 InternVL-78B\n\n## Appendix E Limitati"
|
||||
}
|
||||
],
|
||||
"tables": [
|
||||
{
|
||||
"index": 1,
|
||||
"file": "tables/table-01.txt",
|
||||
"sha256": "bca21eb73b134a03bd6cb2f642c948c1e78b05e5c920ae653a051851af96f224",
|
||||
"rows": 1,
|
||||
"caption": "Figure 1: Overview of our automated three-stage framework for training MM-PRM: supervised policy model construction, MCTS-based process supervision data generation, and step-level reward model training."
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"file": "tables/table-02.txt",
|
||||
"sha256": "14bc24e0a27ecb4b5ca3b097c9208e8c4fb3c7f6fd0bda1273df9057aa422c49",
|
||||
"rows": 1,
|
||||
"caption": "Table 1: Performance improvements across various benchmarks when applying the MM-PRM to different models."
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"file": "tables/table-03.txt",
|
||||
"sha256": "ace610723a7db939b9bcc1bd412a49cd9852c62d185f9ba1b22fe557de3467db",
|
||||
"rows": 1,
|
||||
"caption": "Figure 2: Qualitative example of MM-PRM accurately identifying error steps in multimodal reasoning process."
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"file": "tables/table-04.txt",
|
||||
"sha256": "1af2dadea5b752a154e1b0b5be19dd5f5f7ea1f071be11da2426af4357354599",
|
||||
"rows": 16,
|
||||
"caption": "Figure 3: (a) Effect of the number of candidate reasoning paths on answer accuracy under Best-of-N inference. Increasing the number of candidates allows the PRM to select higher-quality reasoning trajectories. (b) Effect of learning rate on PRM performance. Small learning rates yield better accuracy, with performance peaking at 4e-6"
|
||||
},
|
||||
{
|
||||
"index": 5,
|
||||
"file": "tables/table-05.txt",
|
||||
"sha256": "d2fa38708b8990f98a98431a8b5acc21d968a2768c4297f4310d9e6e5d6780b2",
|
||||
"rows": 3,
|
||||
"caption": "Table 2: The performance comparisons of soft label vs. hard label."
|
||||
},
|
||||
{
|
||||
"index": 6,
|
||||
"file": "tables/table-06.txt",
|
||||
"sha256": "73560adcaaae05b42680a7621b0e9556fa28eab6fdd11785da1a4c4c37c7aa0b",
|
||||
"rows": 18,
|
||||
"caption": "Table 3: Data sources and sample counts used for policy model training."
|
||||
},
|
||||
{
|
||||
"index": 7,
|
||||
"file": "tables/table-07.txt",
|
||||
"sha256": "fadbccdfed3989bf4bbf86b8f9b7194b95323a696da85ebf290ec7e96a7f2381",
|
||||
"rows": 1,
|
||||
"caption": "Table 4: Performance comparison of various aggregators on various benchmarks for MM-Policy. Top performer is in bold."
|
||||
},
|
||||
{
|
||||
"index": 8,
|
||||
"file": "tables/table-08.txt",
|
||||
"sha256": "2902e64333b18054c6ba3ea96a2bf19d6eef8029c70ace540c34a07fcf7bbe97",
|
||||
"rows": 1,
|
||||
"caption": "Table 5: Performance comparison of various aggregators on various benchmarks for InternVL-8B. Top performer is in bold."
|
||||
},
|
||||
{
|
||||
"index": 9,
|
||||
"file": "tables/table-09.txt",
|
||||
"sha256": "17906336b566b81f9ca81e77d0c296b31c6e6dd9d50df1e992a6aa591d6b746a",
|
||||
"rows": 1,
|
||||
"caption": "Table 6: Performance comparison of various aggregators on various benchmarks for InternVL-26B. Top performer is in bold."
|
||||
},
|
||||
{
|
||||
"index": 10,
|
||||
"file": "tables/table-10.txt",
|
||||
"sha256": "77bed1c46fe17f44e239e4666ed821d278f125b282c9b30972bcab1750fcb0b7",
|
||||
"rows": 1,
|
||||
"caption": "Table 7: Performance comparison of various aggregators on various benchmarks for InternVL-38B. Top performer is in bold."
|
||||
},
|
||||
{
|
||||
"index": 11,
|
||||
"file": "tables/table-11.txt",
|
||||
"sha256": "14cf3b9d68af5ea6ff37ecc691df77683368dc5c2d20b5d2c5a519fbc57e67e8",
|
||||
"rows": 1,
|
||||
"caption": "Table 8: Performance comparison of various aggregators on various benchmarks for InternVL-78B. Top performer is in bold."
|
||||
},
|
||||
{
|
||||
"index": 12,
|
||||
"file": "tables/table-12.txt",
|
||||
"sha256": "7eb0253d8b701608e26a1b957480706654e8ee5537ff30f51231d8cdbd52d9e1",
|
||||
"rows": 1,
|
||||
"caption": ""
|
||||
},
|
||||
{
|
||||
"index": 13,
|
||||
"file": "tables/table-13.txt",
|
||||
"sha256": "aeb48c93e516482c221ba9d5d47f24db83065269ad2af37d1f445b294b87b437",
|
||||
"rows": 6,
|
||||
"caption": ""
|
||||
},
|
||||
{
|
||||
"index": 14,
|
||||
"file": "tables/table-14.txt",
|
||||
"sha256": "1c6db792e5eeb1309907a7f6e689d2b4108c6cfc576d77fb01e1d293fa77300d",
|
||||
"rows": 6,
|
||||
"caption": ""
|
||||
},
|
||||
{
|
||||
"index": 15,
|
||||
"file": "tables/table-15.txt",
|
||||
"sha256": "532ca1fa293217c29a44ab85265e0d68b5b6661091b4eded0034f735c8c8b410",
|
||||
"rows": 6,
|
||||
"caption": ""
|
||||
},
|
||||
{
|
||||
"index": 16,
|
||||
"file": "tables/table-16.txt",
|
||||
"sha256": "58d9897acde561ed286172d97f59cb5549e0c18980d9b77c333d284b3d767196",
|
||||
"rows": 6,
|
||||
"caption": ""
|
||||
},
|
||||
{
|
||||
"index": 17,
|
||||
"file": "tables/table-17.txt",
|
||||
"sha256": "6cbc55fd8d522e7a8f6eec62cc3af55514ec78a46bc121f10b8d9f4a66de0c73",
|
||||
"rows": 6,
|
||||
"caption": ""
|
||||
}
|
||||
],
|
||||
"figures": [
|
||||
{
|
||||
"index": 1,
|
||||
"file": "figures/fig-01.png",
|
||||
"url": "https://ar5iv.labs.arxiv.org/html/2505.13427/assets/overview.png",
|
||||
"sha256": "1f589df58cdbc392c3adfe9f22fed636355655a7446a21978bd9f7cdcc4bc474",
|
||||
"bytes": 1177584,
|
||||
"fetch_ok": true,
|
||||
"caption": "Figure 1: Overview of our automated three-stage framework for training MM-PRM: supervised policy model construction, MCTS-based process supervision data generation, and step-level reward model training."
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"file": "figures/fig-02.png",
|
||||
"url": "https://ar5iv.labs.arxiv.org/html/2505.13427/assets/crop_qualitative.png",
|
||||
"sha256": "84560437b6aeefc4064b4e8185ea78e9eff6cc57dcf4897f28499c39d0d5ecf9",
|
||||
"bytes": 179793,
|
||||
"fetch_ok": true,
|
||||
"caption": "Table 1: Performance improvements across various benchmarks when applying the MM-PRM to different models."
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"file": "figures/fig-03.png",
|
||||
"url": "https://ar5iv.labs.arxiv.org/html/2505.13427/assets/combined_plot.png",
|
||||
"sha256": "28b291d92207b402dc386eb48dc4d444221f7a3a79bf5bb949f8d5441b15f58a",
|
||||
"bytes": 600991,
|
||||
"fetch_ok": true,
|
||||
"caption": "Figure 2: Qualitative example of MM-PRM accurately identifying error steps in multimodal reasoning process."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,883 @@
|
||||
[2505.13427] MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
|
||||
|
||||
## MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
|
||||
|
||||
Lingxiao Du1 Fanqing Meng1,3∗ Zongkai Liu2∗ Zhixiang Zhou2∗
|
||||
Ping Luo4 Qiaosheng Zhang2,3† Wenqi Shao2,3
|
||||
1Shanghai AI Laboratory 2Shanghai Innovation Institute
|
||||
3Shanghai Jiao Tong University 4The University of Hong Kong
|
||||
|
||||
Thanks: Equal contribution
|
||||
|
||||
Thanks: Corresponding Authors: shaowenqi@pjlab.org.cn; zhangqiaosheng@pjlab.org.cn
|
||||
|
||||
## Abstract
|
||||
|
||||
While Multimodal Large Language Models (MLLMs) have achieved impressive progress in vision-language understanding, they still struggle with complex multi-step reasoning, often producing logically inconsistent or partially correct solutions. A key limitation lies in the lack of fine-grained supervision over intermediate reasoning steps. To address this, we propose MM-PRM, a process reward model trained within a fully automated, scalable framework. We first build MM-Policy, a strong multimodal model trained on diverse mathematical reasoning data. Then, we construct MM-K12, a curated dataset of 10,000 multimodal math problems with verifiable answers, which serves as seed data. Leveraging a Monte Carlo Tree Search (MCTS)-based pipeline, we generate over 700k step-level annotations without human labeling. The resulting PRM is used to score candidate reasoning paths in the Best-of-N inference setup and achieves significant improvements across both in-domain (MM-K12 test set) and out-of-domain (OlympiadBench, MathVista, etc.) benchmarks. Further analysis confirms the effectiveness of soft labels, smaller learning rates, and path diversity in optimizing PRM performance. MM-PRM demonstrates that process supervision is a powerful tool for enhancing the logical robustness of multimodal reasoning systems. We release all our codes and data at https://github.com/ModalMinds/MM-PRM.
|
||||
|
||||
## 1 Introduction
|
||||
|
||||
The rapid advancement of Large Language Models (LLMs) achiam2023gpt; bai2023qwen; team2023gemini; touvron2023llama; touvron2023llama2; grattafiori2024llama; 2023internlm; cai2024internlm2; guo2025deepseek has significantly improved performance on a wide range of natural language processing tasks, including general reasoning and mathematical problem-solving. In parallel, the development of Multimodal Large Language Models (MLLMs) liu2023visual; yao2024minicpm; chen2024expanding; wang2024mpo; gao2024mini; chen2024far; chen2024internvl; Qwen-VL; team2025kimi has unlocked new capabilities in vision-language understanding, showing promising results in areas such as image captioning and visual question answering (VQA). However, despite their impressive capabilities in perception and basic reasoning, MLLMs still struggle with complex multi-step reasoning tasks, particularly in mathematics. These shortcomings often manifest as broken logical chains, inaccurate intermediate steps, or cases that produce incorrect intermediate steps while still occasionally arriving at the correct final answer—a phenomenon that introduces high false-positive rates and undermines interpretability.
|
||||
|
||||
To address this issue, reward modeling cobbe2021training; yu2023ovm; uesato2022solving; zang2025internlm has emerged as a promising paradigm. Reward models (RMs) play a central role in reinforcement learning from human feedback (RLHF) ouyang2022training; schulman2017proximal; shao2024deepseekmath; lai2024step; wang2024enhancing; pang2024iterative, and can also be used at inference time to select among multiple candidate responses using Test-Time Scaling (TTS) dong2024rlhf; snell2024scaling; wang2023math; zhang2025lessons; feng2023alphazero; kang2024mindstar; ma2023let; tian2024toward; zhang2024rest strategies such as Best-of-N (BoN).
|
||||
In general, reward models for reasoning tasks can be broadly categorized into two types: Outcome Reward Models (ORMs) and Process Reward Models (PRMs). ORMs cobbe2021training; yu2023ovm provide scalar feedback only on the final answer, overlooking the quality of the intermediate reasoning steps. This limits their ability to guide the model toward robust reasoning paths. In contrast, PRMs li2022making; lightman2023let; uesato2022solving; wang2023math; wang2024multi; luo2024improve offer a more fine-grained approach by evaluating each reasoning step, enabling more accurate and interpretable feedback.
|
||||
|
||||
Some recent works have explored process reward models for mathematical reasoning in pure text. PRM800k lightman2023let manually constructs a large-scale dataset with step-level correctness labels, which is hard to scale. MathShepherd wang2023math adopts Monte Carlo estimation to label reasoning steps by evaluating whether continuations from a given step lead to the correct answer, but its efficiency is relatively low. OmegaPRM luo2024improve introduces a Monte Carlo Tree Search (MCTS)-based framework that enables efficient automated generation of process supervision data. However, all these works focus on mathematical reasoning in pure text. In the field of multimodal mathematical reasoning, how to design an efficient framework for process supervision data generation and stably train process reward models remains a challenging problem.
|
||||
|
||||
To address these challenges, we propose MM-PRM, a powerful process reward model that effectively handles both in-domain and out-of-domain problems. Specifically, we first collect a large-scale, high-quality reasoning dataset and refine its format to train a policy model called MM-Policy with strong reasoning abilities and well-structured output. Inspired by OmegaPRM, we then establish an efficient multimodal process annotation framework. We first collect 10k high-quality K-12-level multimodal mathematical reasoning samples, named MM-K12. Starting with only 10k math problems as seed data, our framework automatically generates over 700k step-level annotations without any human supervision. Finally, we train MM-PRM with these data and evaluate its performance on multiple benchmarks using the BoN approach.
|
||||
|
||||
MM-PRM demonstrates strong performance and generalization across multiple benchmarks. Although we only use process data generated from K-12-level math problems for training, MM-PRM achieves remarkable results with MM-Policy on benchmarks such as MathVista that improves from 62.93% to 67.60%. In addition, even though MM-PRM is trained on data produced by MM-Policy, it still delivers competitive results on other models like InternVL2.5-8B and InternVL2.5-78B. For instance, on the self-collected MM-K12 test set, InternVL2.5-8B improves from 27.01% to 37.80%, and on MathVerse, InternVL2.5-78B improves from 50.18% to 54.47%. These results highlight MM-PRM’s ability to generalize both across datasets and across model size, despite being trained on limited and fixed data sources.
|
||||
|
||||
Finally, we provide a detailed discussion on key factors in PRM training, including learning rate and the choice between soft labels and hard labels. We find that the following factors are essential for stable PRM training: (1) using a small learning rate to ensure stable optimization; (2) adopting soft labels instead of hard thresholding to reduce noise and preserve step-wise uncertainty.
|
||||
|
||||
In summary, our contributions are as follows:
|
||||
|
||||
•
|
||||
|
||||
We collect and release MM-K12, a curated multimodal math dataset containing 10,000 seed problems and 500 test problems, all verified to contain unique, checkable answers.
|
||||
|
||||
•
|
||||
|
||||
We build a policy model named MM-Policy using a large corpus of mathematical reasoning data, and develop a fully automated, MCTS-based pipeline for scalable process supervision generation. Leveraging MM-Policy and only 10k seed problems from MM-K12, our framework produces over 700k step-level annotations without human supervision. Based on this, we train MM-PRM, which demonstrates strong performance across multiple benchmarks—for example, boosting accuracy on the MM-K12 test set from 33.92% to 42.80%, on MathVista from 62.93% to 67.60%, and on OlympiadBench from 15.41% to 24.00%.
|
||||
|
||||
•
|
||||
|
||||
We provide a detailed discussion on the settings of key parameters such as the type of process labels and the learning rate. We summarize the key factors for stable PRM training, which can help the community train PRM models more effectively.
|
||||
|
||||
## 2 Related work
|
||||
|
||||
## 2.1 Mathematical reasoning and reward modeling in LLMs
|
||||
|
||||
Mathematical reasoning has become a focal point in evaluating the deep logical abilities of LLMs. cobbe2021training; hendrycks2021measuring. Unlike general language tasks, mathematical problems demand precise multi-step reasoning and logical coherence, motivating approaches like Chain-of-Thought (CoT) promptingwei2022chain and Self-Consistency sampling wang2022self. Supervised fine-tuning (SFT) with structured solution datasets further reinforces reasoning performance, as demonstrated by models like Qwen bai2023qwen, InternLM 2023internlm, and Gemini team2023gemini. More recently, reinforcement learning (RL)-based optimization, exemplified by OpenAI’s o1 jaech2024openai and DeepSeek R1 guo2025deepseek, has emerged as a powerful strategy that surpassing SFT-only baselines.
|
||||
|
||||
However, despite these advances, current LLMs still frequently produce logically inconsistent reasoning steps or false-positive solutions. To address this issue, reward modeling techniques have emerged. Traditional Outcome Reward Models (ORMs)cobbe2021training; yu2023ovm, which assign rewards based solely on the final answer, fail to detect flawed intermediate reasoning. Process Reward Models (PRMs)lightman2023let; wang2023math; wang2024multi; luo2024improve overcome this limitation by explicitly providing step-level supervision, significantly improving logical coherence. Representative works such as PRM800K lightman2023let, MathShepherd wang2023math, and MiPS wang2024multi have demonstrated the effectiveness of PRMs in enhancing mathematical reasoning.
|
||||
|
||||
Overall, PRMs have become an essential approach for improving the robustness and logical consistency of LLM reasoning through fine-grained, step-level evaluation.
|
||||
|
||||
## 2.2 Process supervision data construction
|
||||
|
||||
Training PRMs requires large-scale, high-quality step-level annotations that indicate the correctness of each intermediate reasoning step. However, most publicly available datasets focus solely on final answers, making it challenging to obtain supervision signals for reasoning quality. Existing process supervision construction methods can be broadly categorized into three classes:
|
||||
|
||||
Manual Annotation.
|
||||
The earliest efforts, such as PRM800K lightman2023let, relied on human experts to label each reasoning step. While this yields high-quality annotations, the approach is costly, time-consuming, and difficult to scale—particularly for domains like mathematics where expert knowledge is required.
|
||||
|
||||
Monte Carlo Estimation.
|
||||
Works like MathShepherd wang2023math and MiPS wang2024multi automate supervision by sampling multiple rollouts from each reasoning prefix and estimating step correctness based on the proportion of successful completions. If continuations from a given step often lead to correct answers, that step is considered reliable. This method is simple but suffers from label instability due to high variance in sampled rollouts—especially on long or complex reasoning chains.
|
||||
|
||||
Monte Carlo Tree Search (MCTS).
|
||||
To address these limitations, OmegaPRM luo2024improve introduces a structured, search-based alternative. Using a divide-and-conquer MCTS algorithm, it efficiently identifies the first error in a reasoning chain and constructs a tree of reasoning paths with dynamically updated statistics. This approach improves annotation stability, better handles deep reasoning tasks, and generates large volumes of fine-grained supervision from limited seed data without human input.
|
||||
|
||||
Each method reflects a trade-off between cost, scalability, and label fidelity. Among them, MCTS-based approaches have recently emerged as the most effective strategy for generating robust and scalable process supervision, especially for multi-step mathematical reasoning tasks.
|
||||
|
||||
## 3 Method
|
||||
|
||||
## 3.1 Overview
|
||||
|
||||
We present a scalable and fully automated framework for training PRMs tailored to multimodal mathematical reasoning. Our approach addresses two critical bottlenecks in prior work: the lack of step-level supervision and the inefficiency of data generation in complex reasoning tasks. To overcome these challenges, we introduce a structured three-stage pipeline that enables fine-grained reward modeling without any human annotation.
|
||||
As shown in Figure 1, the framework comprises three interconnected stages as follows:
|
||||
|
||||
(1) Policy Model Construction, where a multimodal policy model is trained to generate high-quality reasoning traces following the CoT paradigm.
|
||||
|
||||
(2) Process Supervision Data Generation, where we use a MCTS-based engine, OmegaPRM luo2024improve, to efficiently identify reasoning flaws and produce step-level reward labels at scale.
|
||||
|
||||
(3) Process Reward Model Training, where a PRM is trained to evaluate each reasoning step and provide dense feedback.
|
||||
|
||||
This end-to-end design ensures that process supervision can be generated, modeled, and applied in a fully closed loop. It significantly improves reasoning quality and robustness, especially in tasks requiring long logical chains. In Sections 3.2–3.3 below, we elaborate on these three stages.
|
||||
|
||||
## 3.2 Policy model construction
|
||||
|
||||
The policy model serves as the foundation of our framework, responsible for generating candidate reasoning trajectories given multimodal math problems. These trajectories are later evaluated and labeled to form step-level supervision for training the PRM. Ensuring that the policy model produces logically coherent and structurally complete outputs is thus essential for the effectiveness of the entire system.
|
||||
|
||||
To train the policy model, we curated a large-scale, high-quality dataset of mathematical problems spanning a wide range of topics and difficulty levels. Our dataset integrates samples from over a dozen public math datasets, including R-CoT deng2024r, MAVIS zhang2024mavis, MathV360K shi2024math, NuminaMath li2024numinamath, and DART-Math tong2024dart, with problems ranging from elementary school arithmetic to advanced geometry and statistics. The full list of data sources and sample counts used for training is provided in Appendix A.
|
||||
|
||||
Once collected, all data undergo rigorous cleaning and format standardization. Visual and textual content are paired explicitly, and reasoning traces are reformatted to follow a structured CoT schema, with each logical step clearly marked using structured tags such as <step></step>, and the final conclusion labeled with <answer></answer>. To enhance quality and clarity, we leveraged a strong instruction-tuned language model (i.e., Qwen2.5-72B-Instruct bai2023qwen) to parse original solutions and restructure them into coherent, modular steps. This structured representation not only enhances model learnability but also lays the foundation for generating step-level reward labels in the next stage. The prompt used for solution restructuring is provided in Appendix B.
|
||||
|
||||
With this cleaned and annotated corpus of over 5 million examples, we fine-tuned a strong open-source multimodal model, InternVL2.5-8B chen2024internvl, using supervised learning. This ensures that the model learns to produce logically sound and well-structured outputs that conform to the CoT reasoning pattern.
|
||||
|
||||
The resulting model not only delivers high-quality reasoning trajectories for downstream data annotation but also supports inference-time use cases such as BoN selection with reward-based reranking. Its strong generative ability and logical consistency form the cornerstone of our scalable process supervision framework.
|
||||
|
||||
## 3.3 Process supervision data generation
|
||||
|
||||
To enable fine-grained supervision for step-level reasoning, we adopt an automated process annotation pipeline based on the OmegaPRM luo2024improve framework. OmegaPRM introduces a MCTS-based mechanism for efficiently identifying and labeling intermediate reasoning steps with confidence estimates. Although originally developed for textual mathematical reasoning, we adapt and extend this framework to handle multimodal inputs.
|
||||
|
||||
Our process begins with collecting a curated dataset of 10,000 multimodal math problems from real-world named MM-K12—consisting of 5,000 fill-in-the-blank and 5,000 multiple-choice questions. These problems span a range of curriculum topics from elementary to high school and serve as seed instances for process supervision generation. All examples in MM-K12 are carefully filtered to ensure that each question includes meaningful visual input and has a unique, verifiable answer, making them well-suited for structured reasoning and reward modeling. In addition, MM-K12 also provides an independent test set of 500 problems, constructed under the same criteria, which we use to evaluate in-distribution performance later. For each problem, the policy model produces multiple candidate solutions following the CoT paradigm, and these reasoning paths form the raw material for subsequent reward annotation.
|
||||
|
||||
To evaluate the correctness of each intermediate step, we follow OmegaPRM’s hierarchical rollout and search protocol. Specifically, we generate multiple completions (rollouts) from partial prefixes and estimate the correctness of a given step based on whether its downstream completions reach the correct final answer. By applying binary search, the algorithm efficiently pinpoints the earliest step at which the reasoning begins to deviate. These supervision signals are then aggregated within a structured state-action tree, which records the Monte Carlo (MC) estimates and other statistics at each reasoning state. In our implementation, we maintain the full multimodal context—including both textual and visual components—throughout the tree construction and search process.
|
||||
|
||||
Importantly, our adaptation retains the efficiency of OmegaPRM’s divide-and-conquer search while enabling reward supervision for reasoning steps that are conditioned on complex visual stimuli. Through this pipeline, we generate over 700,000 step-level annotations from only 10k seed questions, without requiring manual labeling. The resulting dataset provides dense, high-quality process supervision aligned with real-world multimodal reasoning.
|
||||
|
||||
## 3.4 Process reward model training
|
||||
|
||||
With large-scale step-level supervision in place, we proceed to train a PRM that can assess the quality of reasoning steps given a multimodal context. The PRM is designed to serve as a fine-grained critic, assigning a reward score to each intermediate step conditioned on its preceding reasoning context that enables both test-time scaling and potential RL applications.
|
||||
|
||||
## 3.4.1 Labeling strategy
|
||||
|
||||
A central design decision in PRM training lies in how to formulate supervision signals from the MC estimations luo2024improve; zhang2025lessons. Rather than adopting hard binary label (e.g., y^=1[MC(s)>τ]\hat{y}=1[\mathrm{MC}(s)>\tau]), we use soft label, directly taking the MC scores as continuous supervision targets.
|
||||
|
||||
This choice is motivated by the observation that the MC score reflects more than the correctness of an intermediate step. It also encodes factors such as problem difficulty, step criticality, and distributional uncertainty in the policy model’s rollouts. For instance, a reasoning step within a highly ambiguous or visually complex problem may yield lower MC scores even if the logic is fundamentally sound. In such cases, hard-thresholding may misrepresent the step’s quality, introducing noise into training. By contrast, soft labels preserve the probabilistic nuance and enable smoother learning dynamics, more details will be discussed in section 5.3
|
||||
|
||||
Formally, for each reasoning step xtx_{t} in a path x=[x1,x2,…,xT]x=[x_{1},x_{2},\dots,x_{T}], we assign a supervision target y^t=MC(x<t)∈[0,1]\hat{y}_{t}=\mathrm{MC}(x_{<t})\in[0,1], where MC(x<t)\mathrm{MC}(x_{<t}) denotes the estimated probability that a correct final answer can be reached from this partial path.
|
||||
|
||||
## 3.4.2 Model design and training objective
|
||||
|
||||
To model the prediction task, we treat the PRM as a classifier operating on each reasoning step. Given a multimodal input qq and a generated reasoning trace [x1,x2,…,xT][x_{1},x_{2},\dots,x_{T}], we interleave a special marker token, denoted σ\sigma, after each step, producing an input sequence of the form:
|
||||
|
||||
In our implementation, σ\sigma is instantiated as the token <prm>. At each occurrence of σ\sigma, the model is tasked with producing a scalar confidence score indicating the likelihood that the immediately preceding step is logically correct. Formally, let zYes(i)z^{(i)}_{\texttt{Yes}} and zNo(i)z^{(i)}_{\texttt{No}} denote the unnormalized logits for binary labels “Yes” (correct) and “No” (incorrect) at the ii-th occurrence of σ\sigma. The model’s predicted probability is computed via softmax:
|
||||
|
||||
The training objective is to minimize the cross-entropy loss between the predicted scores p(i)p^{(i)} and the soft labels y^(i)\hat{y}^{(i)}, across all scoring points:
|
||||
|
||||
This formulation guides the model to make fine-grained assessments of reasoning steps, assigning higher confidence to those with stronger evidence of correctness.
|
||||
|
||||
## 4 Experiments
|
||||
|
||||
## 4.1 Experiments setup
|
||||
|
||||
To validate the effectiveness of our proposed process reward modeling framework, we conduct a series of experiments, carefully configured to ensure fair, scalable, and reproducible results.
|
||||
|
||||
Policy model construction.
|
||||
Our policy model(i.e., MM-Policy) is initialized from the multimodal backbone InternVL 2.5-8B and fine-tuned using approximately 4 million cleaned, structured math problems. The model is trained for 1 epoch with a batch size of 128 and a learning rate of 4e-5, updating only the language module while keeping the vision encoder frozen.
|
||||
|
||||
Process supervision data generation.
|
||||
We adapt the OmegaPRMluo2024improve pipeline for multimodal reasoning and apply it to MM-K12 (10k samples). Using MCTS-based structured rollouts, we generate approximately 747,000 step-level annotations. The sampling parameters are tuned for balance between diversity and efficiency: temperature=1.0temperature=1.0, topk=50top_{k}=50, topp=0.9top_{p}=0.9, exploration coefficient cpuct=0.125c_{\text{puct}}=0.125, and up to 200 search steps or 1,000 total rollouts per problem.
|
||||
|
||||
Process reward model training.
|
||||
We initialize the PRM from the fine-tuned policy model and train it for 1 epochs with a batch size of 512 and a learning rate of 4e-6.
|
||||
|
||||
## 4.2 Evaluation strategies and benchmarks
|
||||
|
||||
To assess the effectiveness of the proposed MM-PRM in improving reasoning quality, we adopt the BoN evaluation protocol. For each test problem, the policy model generates N=16N=16 candidate reasoning paths independently. The PRM then scores each path step-by-step, producing a sequence of floating-point values representing the predicted quality of each intermediate step, the path with the highest score is selected as the final answer.
|
||||
|
||||
Since PRM outputs a vector of step-wise confidence scores for each candidate path, a crucial component of our evaluation is the aggregation function wang2024multi used to compress this vector into a scalar. We explore a diverse set of aggregation functions, including Min, Average, Max, SumLogPr (i.e., sum of log-probabilities), SumLogOdds (i.e., sum of log-odds), and MeanOdds (i.e., mean odds), each function captures different aspects of path quality. Additionally, a Random baseline is used for comparison, where the final answer is randomly sampled from the same set of 16 candidates. Formal definitions of all aggregation functions are provided in Appendix C.
|
||||
|
||||
We evaluate performance using answer accuracy, defined as the proportion of final selected answers that match the ground truth. This metric directly reflects MM-PRM’s utility in guiding the selection of correct reasoning paths. To comprehensively evaluate our model’s performance and generalization, we conduct experiments on a range of multimodal math benchmarks, including MM-K12 (test set), OlympiadBench (OE_MM_maths_en_COMP) he2024olympiadbench, MathVista(testmini) lu2023mathvista, MathVerse(testmini) zhang2024mathverse, and MathVision(test) wang2024measuring. The MM-K12 test set serves as an in-distribution evaluation. For out-of-distribution assessment, we use the OE_MM_maths_en_COMP split of OlympiadBench, which contains open-ended multimodal questions from international math competitions, closely related in format to MM-K12 but significantly harder. To further test generalization, we include MathVista, which covers a wide range of visual mathematical tasks; MathVerse, which emphasizes understanding structured visual content; and MathVision, which targets abstract visual reasoning. These benchmarks provide a diverse and rigorous setting to measure both performance and generalization of our process reward modeling framework.
|
||||
|
||||
## 4.3 Quantitative results
|
||||
|
||||
We evaluate the effectiveness of MM-PRM by applying it to a range of policy models and testing its impact across multiple multimodal math benchmarks.
|
||||
|
||||
Across all models, MM-PRM yields substantial performance improvements. For example, when applied to MM-Policy on the MM-K12 test set, accuracy improves from 33.92% to 42.80%, and similar gains are observed with InternVL2.5-8B, where performance rises from 27.01% to 37.80%. These results confirm that MM-PRM is highly effective at identifying high-quality reasoning paths.
|
||||
|
||||
Beyond in-domain settings, we observe that MM-PRM also generalizes well to larger models and more challenging datasets. As shown in Table 1, applying MM-PRM to InternVL2.5-78B improves accuracy on OlympiadBench from 30.98% to 34.67%, and on MathVerse from 50.18% to 54.47%. Despite being trained only on the small MM-K12 seed dataset and with a fixed policy model(i.e., MM-Policy), MM-PRM consistently enhances reasoning accuracy across diverse benchmarks and models. This demonstrates the potential of scalable step-level reward modeling to improve mathematical reasoning in a model-agnostic and data-efficient manner. Detailed evaluation results across all aggregation functions are provided in Appendix D.
|
||||
|
||||
## 4.4 Qualitative results
|
||||
|
||||
To further illustrate the behavior of MM-PRM, we present a qualitative example involving a geometry problem with parallel lines and an angle bisector. As shown in Figure 2, the policy model generates a four-step reasoning path, which ultimately leads to an incorrect final answer.
|
||||
|
||||
The PRM assigns high scores to the first two steps (0.83 and 0.68), indicating that they are logically sound. In contrast, Step-3 receives a very low score (0.02), signaling that the model has identified a significant reasoning error at this point. This flawed step leads directly to an incorrect conclusion in Step-4.
|
||||
|
||||
This example demonstrates that MM-PRM is capable of detecting localized logical errors within a reasoning chain, such fine-grained judgment is crucial in selecting high-quality reasoning trajectories and filtering out those with subtle but critical flaws.
|
||||
|
||||
## 5 Discussion
|
||||
|
||||
## 5.1 Candidate path’s impact on PRM performance
|
||||
|
||||
Since the PRM operates purely as a selector in the BoN framework, its performance is inherently bounded by the diversity and quality of candidate reasoning paths produced by the policy model. In other words, PRM cannot improve a flawed generation in BoN—it can only choose among the available options. Therefore, the number of reasoning paths generated per problem directly affects its potential to identify a correct and coherent solution.
|
||||
|
||||
To study this effect, we vary the number of generated reasoning paths N∈{2,22,23,24}N\in\{2,2^{2},2^{3},2^{4}\} and measure the corresponding answer accuracy under the aggregation strategy MeanOdds. As shown in Figure 3(a), increasing NN consistently improves MM-PRM’s performance across both test sets.
|
||||
On the MM-K12 test set, accuracy improves from 38.6% at N=2N=2 to 42.8% at N=16N=16, with gains tapering off beyond N=8N=8.
|
||||
In contrast, on OlympiadBench, accuracy increases more steadily—from 18.4% to 24.0%—as NN grows. This suggests that for harder, more diverse tasks, having a larger pool of reasoning paths is critical for PRM to identify valid solutions.
|
||||
|
||||
## 5.2 Learning rate
|
||||
|
||||
As noted in lightman2023let, finetuning a PRM shifts the language models’ objective from generation to discrimination, making learning rate a critical factor. Smaller learning rates are often preferred to maintain stability and preserve pretrained knowledge.
|
||||
|
||||
We evaluate MM-PRM trained under different learning rates on the MM-K12 test set using MeanOdds aggregator. As shown in Figure 3(b), performance peaks at 4e-6—about one-tenth the learning rate typically used in supervised fine-tuning—then drops sharply at higher values. This confirms that a moderate, conservative learning rate leads to better training, while overly large values degrade accuracy.
|
||||
|
||||
## 5.3 Soft label vs. Hard label
|
||||
|
||||
As discussed in section 3.4.1, we adopt soft label—i.e., real-valued MC scores—as supervision for step-level reward modeling. Unlike hard label, soft label retain uncertainty and allow the model to learn more nuanced representations of reasoning quality.
|
||||
|
||||
To assess this design choice, we compare soft label with hard label thresholding, where steps with MC>0MC>0 are treated as correct, and others as incorrect, following the protocol in luo2024improve; wang2024openr.
|
||||
|
||||
As shown in Table 2, soft-label training consistently outperforms hard-label training across all aggregation strategies. For instance, under the Average aggregator, soft labels yield 43% accuracy on MM-K12 test set, compared to 34.4% with hard labels. Similar improvements are observed with SumLogOdds (43.2% vs. 33.8%) and MeanOdds (42.8% vs. 37.0%).
|
||||
|
||||
## 6 Conclusion
|
||||
|
||||
In this work, we introduce MM-PRM, a process reward model built upon a scalable framework for multimodal mathematical process reward modeling that enables step-level supervision without human annotation. By leveraging a multimodal policy model and an MCTS-based data generation pipeline, we construct over 700k process-level labels from only 10k K-12 math problems in our MM-K12 dataset. Our trained PRM significantly improves reasoning accuracy in BoN inference across both standard and challenging benchmarks, and demonstrates strong generalization to new datasets and models. Extensive analysis further confirms the importance of soft labeling, conservative learning rates, and sufficient path diversity. MM-PRM highlights the value of process supervision for enhancing multimodal mathematical problem-solving.
|
||||
|
||||
## References
|
||||
|
||||
[1]
|
||||
|
||||
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya,
|
||||
Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman,
|
||||
Shyamal Anadkat, et al.
|
||||
|
||||
Gpt-4 technical report.
|
||||
|
||||
arXiv preprint arXiv:2303.08774, 2023.
|
||||
|
||||
[2]
|
||||
|
||||
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan,
|
||||
Wenbin Ge, Yu Han, Fei Huang, et al.
|
||||
|
||||
Qwen technical report.
|
||||
|
||||
arXiv preprint arXiv:2309.16609, 2023.
|
||||
|
||||
[3]
|
||||
|
||||
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu,
|
||||
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican,
|
||||
et al.
|
||||
|
||||
Gemini: a family of highly capable multimodal models.
|
||||
|
||||
arXiv preprint arXiv:2312.11805, 2023.
|
||||
|
||||
[4]
|
||||
|
||||
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
|
||||
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
|
||||
Hambro, Faisal Azhar, et al.
|
||||
|
||||
Llama: Open and efficient foundation language models.
|
||||
|
||||
arXiv preprint arXiv:2302.13971, 2023.
|
||||
|
||||
[5]
|
||||
|
||||
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
|
||||
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
|
||||
et al.
|
||||
|
||||
Llama 2: Open foundation and fine-tuned chat models.
|
||||
|
||||
arXiv preprint arXiv:2307.09288, 2023.
|
||||
|
||||
[6]
|
||||
|
||||
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek
|
||||
Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex
|
||||
Vaughan, et al.
|
||||
|
||||
The llama 3 herd of models.
|
||||
|
||||
arXiv preprint arXiv:2407.21783, 2024.
|
||||
|
||||
[7]
|
||||
|
||||
InternLM Team.
|
||||
|
||||
Internlm: A multilingual language model with progressively enhanced
|
||||
capabilities.
|
||||
|
||||
https://github.com/InternLM/InternLM-techreport, 2023.
|
||||
|
||||
[8]
|
||||
|
||||
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen,
|
||||
Zehui Chen, Zhi Chen, Pei Chu, et al.
|
||||
|
||||
Internlm2 technical report.
|
||||
|
||||
arXiv preprint arXiv:2403.17297, 2024.
|
||||
|
||||
[9]
|
||||
|
||||
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu,
|
||||
Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
|
||||
|
||||
Deepseek-r1: Incentivizing reasoning capability in llms via
|
||||
reinforcement learning.
|
||||
|
||||
arXiv preprint arXiv:2501.12948, 2025.
|
||||
|
||||
[10]
|
||||
|
||||
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
|
||||
|
||||
Visual instruction tuning.
|
||||
|
||||
Advances in neural information processing systems,
|
||||
36:34892–34916, 2023.
|
||||
|
||||
[11]
|
||||
|
||||
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi
|
||||
Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al.
|
||||
|
||||
Minicpm-v: A gpt-4v level mllm on your phone.
|
||||
|
||||
arXiv preprint arXiv:2408.01800, 2024.
|
||||
|
||||
[12]
|
||||
|
||||
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo
|
||||
Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
|
||||
|
||||
Expanding performance boundaries of open-source multimodal models
|
||||
with model, data, and test-time scaling.
|
||||
|
||||
arXiv preprint arXiv:2412.05271, 2024.
|
||||
|
||||
[13]
|
||||
|
||||
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo
|
||||
Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai.
|
||||
|
||||
Enhancing the reasoning ability of multimodal large language models
|
||||
via mixed preference optimization.
|
||||
|
||||
arXiv preprint arXiv:2411.10442, 2024.
|
||||
|
||||
[14]
|
||||
|
||||
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao
|
||||
Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al.
|
||||
|
||||
Mini-internvl: a flexible-transfer pocket multi-modal model with 5%
|
||||
parameters and 90% performance.
|
||||
|
||||
Visual Intelligence, 2(1):1–17, 2024.
|
||||
|
||||
[15]
|
||||
|
||||
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen
|
||||
Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al.
|
||||
|
||||
How far are we to gpt-4v? closing the gap to commercial multimodal
|
||||
models with open-source suites.
|
||||
|
||||
Science China Information Sciences, 67(12):220101, 2024.
|
||||
|
||||
[16]
|
||||
|
||||
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong,
|
||||
Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.
|
||||
|
||||
Internvl: Scaling up vision foundation models and aligning for
|
||||
generic visual-linguistic tasks.
|
||||
|
||||
In Proceedings of the IEEE/CVF Conference on Computer Vision and
|
||||
Pattern Recognition, pages 24185–24198, 2024.
|
||||
|
||||
[17]
|
||||
|
||||
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang
|
||||
Lin, Chang Zhou, and Jingren Zhou.
|
||||
|
||||
Qwen-vl: A versatile vision-language model for understanding,
|
||||
localization, text reading, and beyond.
|
||||
|
||||
arXiv preprint arXiv:2308.12966, 2023.
|
||||
|
||||
[18]
|
||||
|
||||
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng
|
||||
Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al.
|
||||
|
||||
Kimi k1. 5: Scaling reinforcement learning with llms.
|
||||
|
||||
arXiv preprint arXiv:2501.12599, 2025.
|
||||
|
||||
[19]
|
||||
|
||||
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
|
||||
Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
|
||||
et al.
|
||||
|
||||
Training verifiers to solve math word problems.
|
||||
|
||||
arXiv preprint arXiv:2110.14168, 2021.
|
||||
|
||||
[20]
|
||||
|
||||
Fei Yu, Anningzhe Gao, and Benyou Wang.
|
||||
|
||||
Ovm, outcome-supervised value models for planning in mathematical
|
||||
reasoning.
|
||||
|
||||
arXiv preprint arXiv:2311.09724, 2023.
|
||||
|
||||
[21]
|
||||
|
||||
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa
|
||||
Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.
|
||||
|
||||
Solving math word problems with process-and outcome-based feedback.
|
||||
|
||||
arXiv preprint arXiv:2211.14275, 2022.
|
||||
|
||||
[22]
|
||||
|
||||
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding,
|
||||
Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al.
|
||||
|
||||
Internlm-xcomposer2. 5-reward: A simple yet effective multi-modal
|
||||
reward model.
|
||||
|
||||
arXiv preprint arXiv:2501.12368, 2025.
|
||||
|
||||
[23]
|
||||
|
||||
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
|
||||
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
|
||||
|
||||
Training language models to follow instructions with human feedback.
|
||||
|
||||
Advances in neural information processing systems,
|
||||
35:27730–27744, 2022.
|
||||
|
||||
[24]
|
||||
|
||||
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
|
||||
|
||||
Proximal policy optimization algorithms.
|
||||
|
||||
arXiv preprint arXiv:1707.06347, 2017.
|
||||
|
||||
[25]
|
||||
|
||||
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei
|
||||
Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
|
||||
|
||||
Deepseekmath: Pushing the limits of mathematical reasoning in open
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2402.03300, 2024.
|
||||
|
||||
[26]
|
||||
|
||||
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia.
|
||||
|
||||
Step-dpo: Step-wise preference optimization for long-chain reasoning
|
||||
of llms.
|
||||
|
||||
arXiv preprint arXiv:2406.18629, 2024.
|
||||
|
||||
[27]
|
||||
|
||||
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo
|
||||
Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al.
|
||||
|
||||
Enhancing the reasoning ability of multimodal large language models
|
||||
via mixed preference optimization.
|
||||
|
||||
arXiv preprint arXiv:2411.10442, 2024.
|
||||
|
||||
[28]
|
||||
|
||||
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar,
|
||||
and Jason Weston.
|
||||
|
||||
Iterative reasoning preference optimization.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:116617–116637, 2024.
|
||||
|
||||
[29]
|
||||
|
||||
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan
|
||||
Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang.
|
||||
|
||||
Rlhf workflow: From reward modeling to online rlhf.
|
||||
|
||||
arXiv preprint arXiv:2405.07863, 2024.
|
||||
|
||||
[30]
|
||||
|
||||
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
|
||||
|
||||
Scaling llm test-time compute optimally can be more effective than
|
||||
scaling model parameters.
|
||||
|
||||
arXiv preprint arXiv:2408.03314, 2024.
|
||||
|
||||
[31]
|
||||
|
||||
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu,
|
||||
and Zhifang Sui.
|
||||
|
||||
Math-shepherd: Verify and reinforce llms step-by-step without human
|
||||
annotations.
|
||||
|
||||
arXiv preprint arXiv:2312.08935, 2023.
|
||||
|
||||
[32]
|
||||
|
||||
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu,
|
||||
Dayiheng Liu, Jingren Zhou, and Junyang Lin.
|
||||
|
||||
The lessons of developing process reward models in mathematical
|
||||
reasoning.
|
||||
|
||||
arXiv preprint arXiv:2501.07301, 2025.
|
||||
|
||||
[33]
|
||||
|
||||
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan
|
||||
Zhang, and Jun Wang.
|
||||
|
||||
Alphazero-like tree-search can guide large language model decoding
|
||||
and training.
|
||||
|
||||
arXiv preprint arXiv:2309.17179, 2023.
|
||||
|
||||
[34]
|
||||
|
||||
Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong
|
||||
Li, Xu He, Quan He, Feng Wen, et al.
|
||||
|
||||
Mindstar: Enhancing math reasoning in pre-trained llms at inference
|
||||
time.
|
||||
|
||||
arXiv preprint arXiv:2405.16265, 2024.
|
||||
|
||||
[35]
|
||||
|
||||
Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and
|
||||
Hongxia Yang.
|
||||
|
||||
Let’s reward step by step: Step-level reward model as the navigators
|
||||
for reasoning.
|
||||
|
||||
arXiv preprint arXiv:2310.10080, 2023.
|
||||
|
||||
[36]
|
||||
|
||||
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi,
|
||||
and Dong Yu.
|
||||
|
||||
Toward self-improvement of llms via imagination, searching, and
|
||||
criticizing.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:52723–52748, 2024.
|
||||
|
||||
[37]
|
||||
|
||||
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang.
|
||||
|
||||
Rest-mcts*: Llm self-training via process reward guided tree search.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:64735–64772, 2024.
|
||||
|
||||
[38]
|
||||
|
||||
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and
|
||||
Weizhu Chen.
|
||||
|
||||
Making large language models better reasoners with step-aware
|
||||
verifier.
|
||||
|
||||
arXiv preprint arXiv:2206.02336, 2022.
|
||||
|
||||
[39]
|
||||
|
||||
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker,
|
||||
Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
|
||||
|
||||
Let’s verify step by step.
|
||||
|
||||
In The Twelfth International Conference on Learning
|
||||
Representations, 2023.
|
||||
|
||||
[40]
|
||||
|
||||
Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and
|
||||
Jingbo Shang.
|
||||
|
||||
Multi-step problem solving through a verifier: An empirical analysis
|
||||
on model-induced process supervision.
|
||||
|
||||
arXiv preprint arXiv:2402.02658, 2024.
|
||||
|
||||
[41]
|
||||
|
||||
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara,
|
||||
Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al.
|
||||
|
||||
Improve mathematical reasoning in language models by automated
|
||||
process supervision.
|
||||
|
||||
arXiv preprint arXiv:2406.06592, 2024.
|
||||
|
||||
[42]
|
||||
|
||||
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
|
||||
Tang, Dawn Song, and Jacob Steinhardt.
|
||||
|
||||
Measuring mathematical problem solving with the math dataset.
|
||||
|
||||
arXiv preprint arXiv:2103.03874, 2021.
|
||||
|
||||
[43]
|
||||
|
||||
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
|
||||
Le, Denny Zhou, et al.
|
||||
|
||||
Chain-of-thought prompting elicits reasoning in large language
|
||||
models.
|
||||
|
||||
Advances in neural information processing systems,
|
||||
35:24824–24837, 2022.
|
||||
|
||||
[44]
|
||||
|
||||
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang,
|
||||
Aakanksha Chowdhery, and Denny Zhou.
|
||||
|
||||
Self-consistency improves chain of thought reasoning in language
|
||||
models.
|
||||
|
||||
arXiv preprint arXiv:2203.11171, 2022.
|
||||
|
||||
[45]
|
||||
|
||||
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden
|
||||
Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
|
||||
|
||||
Openai o1 system card.
|
||||
|
||||
arXiv preprint arXiv:2412.16720, 2024.
|
||||
|
||||
[46]
|
||||
|
||||
Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang,
|
||||
Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, et al.
|
||||
|
||||
R-cot: Reverse chain-of-thought problem generation for geometric
|
||||
reasoning in large multimodal models.
|
||||
|
||||
arXiv preprint arXiv:2410.17885, 2024.
|
||||
|
||||
[47]
|
||||
|
||||
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang,
|
||||
Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al.
|
||||
|
||||
Mavis: Mathematical visual instruction tuning with an automatic data
|
||||
engine.
|
||||
|
||||
arXiv preprint arXiv:2407.08739, 2024.
|
||||
|
||||
[48]
|
||||
|
||||
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong
|
||||
Bing, and Roy Ka-Wei Lee.
|
||||
|
||||
Math-llava: Bootstrapping mathematical reasoning for multimodal large
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2406.17294, 2024.
|
||||
|
||||
[49]
|
||||
|
||||
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi
|
||||
Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al.
|
||||
|
||||
Numinamath: The largest public dataset in ai4maths with 860k pairs of
|
||||
competition math problems and solutions.
|
||||
|
||||
Hugging Face repository, 13:9, 2024.
|
||||
|
||||
[50]
|
||||
|
||||
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He.
|
||||
|
||||
Dart-math: Difficulty-aware rejection tuning for mathematical
|
||||
problem-solving.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:7821–7846, 2024.
|
||||
|
||||
[51]
|
||||
|
||||
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen,
|
||||
Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
|
||||
|
||||
Olympiadbench: A challenging benchmark for promoting agi with
|
||||
olympiad-level bilingual multimodal scientific problems.
|
||||
|
||||
arXiv preprint arXiv:2402.14008, 2024.
|
||||
|
||||
[52]
|
||||
|
||||
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh
|
||||
Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
|
||||
|
||||
Mathvista: Evaluating mathematical reasoning of foundation models in
|
||||
visual contexts.
|
||||
|
||||
arXiv preprint arXiv:2310.02255, 2023.
|
||||
|
||||
[53]
|
||||
|
||||
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu,
|
||||
Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al.
|
||||
|
||||
Mathverse: Does your multi-modal llm truly see the diagrams in visual
|
||||
math problems?
|
||||
|
||||
In European Conference on Computer Vision, pages 169–186.
|
||||
Springer, 2024.
|
||||
|
||||
[54]
|
||||
|
||||
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie
|
||||
Zhan, and Hongsheng Li.
|
||||
|
||||
Measuring multimodal mathematical reasoning with math-vision dataset.
|
||||
|
||||
Advances in Neural Information Processing Systems,
|
||||
37:95095–95169, 2024.
|
||||
|
||||
[55]
|
||||
|
||||
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong,
|
||||
Yan Song, Lei Chen, Lionel M Ni, et al.
|
||||
|
||||
Openr: An open source framework for advanced reasoning with large
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2410.09671, 2024.
|
||||
|
||||
[56]
|
||||
|
||||
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang.
|
||||
|
||||
Multimath: Bridging visual and mathematical reasoning for large
|
||||
language models.
|
||||
|
||||
arXiv preprint arXiv:2409.00147, 2024.
|
||||
|
||||
[57]
|
||||
|
||||
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and
|
||||
Qi Liu.
|
||||
|
||||
Multimodal arxiv: A dataset for improving scientific comprehension of
|
||||
large vision-language models.
|
||||
|
||||
arXiv preprint arXiv:2403.00231, 2024.
|
||||
|
||||
[58]
|
||||
|
||||
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng.
|
||||
|
||||
Math-puma: Progressive upward multimodal alignment to enhance
|
||||
mathematical reasoning.
|
||||
|
||||
In Proceedings of the AAAI Conference on Artificial
|
||||
Intelligence, volume 39, pages 26183–26191, 2025.
|
||||
|
||||
[59]
|
||||
|
||||
Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan,
|
||||
Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, et al.
|
||||
|
||||
Multimodal self-instruct: Synthetic abstract image and visual
|
||||
reasoning instruction using language model.
|
||||
|
||||
arXiv preprint arXiv:2407.07053, 2024.
|
||||
|
||||
[60]
|
||||
|
||||
Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li,
|
||||
Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang,
|
||||
Yu Qiao, and Jifeng Dai.
|
||||
|
||||
Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024.
|
||||
|
||||
[61]
|
||||
|
||||
Cyrile Delestre Tom Agonnoude, 2024.
|
||||
|
||||
[62]
|
||||
|
||||
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and
|
||||
Hannaneh Hajishirzi.
|
||||
|
||||
Mathqa: Towards interpretable math word problem solving with
|
||||
operation-based formalisms.
|
||||
|
||||
arXiv preprint arXiv:1905.13319, 2019.
|
||||
|
||||
[63]
|
||||
|
||||
PawanKrd.
|
||||
|
||||
math-gpt-4o-200k.
|
||||
|
||||
Hugging Face repository, 2024.
|
||||
|
||||
[64]
|
||||
|
||||
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and
|
||||
Wenhu Chen.
|
||||
|
||||
Mammoth: Building math generalist models through hybrid instruction
|
||||
tuning.
|
||||
|
||||
arXiv preprint arXiv:2309.05653, 2023.
|
||||
|
||||
## Appendix
|
||||
|
||||
## Appendix A Policy Model training data
|
||||
|
||||
Table 3 summarizes the datasets and sample counts used to train the policy model.
|
||||
|
||||
## Appendix B Prompt template for data cleaning
|
||||
|
||||
The prompt template used for data cleaning is shown below. Before generating logical steps and the final answer, the language model is first instructed to explicitly identify the key knowledge points involved in the problem, along with brief explanations. This two-stage prompting strategy enhances the model’s understanding of the question prior to solution restructuring.
|
||||
|
||||
## Appendix C Aggregation function definitions
|
||||
|
||||
In the BoN inference setup, each reasoning path is assigned a vector of step-level scores predicted by the PRM. To compare and rank these paths, we apply an aggregation function to compress each score vector into a single scalar value, following the design considerations discussed in wang2024multi. The following aggregation strategies are used in our experiments:
|
||||
|
||||
•
|
||||
|
||||
Min: Computes the minimum of all step scores:
|
||||
|
||||
•
|
||||
|
||||
Max: Computes the maximum of all step scores:
|
||||
|
||||
•
|
||||
|
||||
Average: Computes the arithmetic mean of all step scores:
|
||||
|
||||
•
|
||||
|
||||
SumLogPr: Computes the sum of log-probabilities of all step scores:
|
||||
|
||||
•
|
||||
|
||||
SumLogOdds: Computes the sum of log-odds of all step scores:
|
||||
|
||||
•
|
||||
|
||||
MeanOdds: Computes the mean of odds-transformed step scores:
|
||||
|
||||
Here, rjr_{j} denotes the jj-th reasoning path consisting of TjT_{j} steps, and pi(j)∈(0,1)p_{i}^{(j)}\in(0,1) is the predicted correctness probability of the ii-th step in that path.
|
||||
|
||||
## Appendix D Full evaluation results
|
||||
|
||||
## D.1 MM-Policy
|
||||
|
||||
## D.2 InternVL-8B
|
||||
|
||||
## D.3 InternVL-26B
|
||||
|
||||
## D.4 InternVL-38B
|
||||
|
||||
## D.5 InternVL-78B
|
||||
|
||||
## Appendix E Limitations
|
||||
|
||||
While our proposed framework demonstrates strong performance and generalization across multiple benchmarks, it also has several limitations: (1) Due to computational constraints, we conduct training only on the InternVL series with 8B parameters, without exploring larger models or architectures from other model families. This restricts our ability to fully assess how PRM training behavior scales with model size or generalizes across different backbones. (2) The seed data used for process supervision generation is limited in diversity, as it consists solely of K-12 math problems. As a result, the PRM may be less exposed to advanced mathematical domains or visual formats beyond the scope of standard educational settings. We leave broader model coverage and more diverse seed data construction as promising directions for future work.
|
||||
|
||||
◄
|
||||
|
||||
Feeling
|
||||
lucky?
|
||||
|
||||
Conversion
|
||||
report
|
||||
Report
|
||||
an issue
|
||||
View original
|
||||
on arXiv►
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1 @@
|
||||
| [q,x1,σ,x2,σ,…,xT,σ].[q,x_{1},\sigma,x_{2},\sigma,\dots,x_{T},\sigma]. |
|
||||
@@ -0,0 +1 @@
|
||||
| p(i)=exp(zYes(i))exp(zYes(i))+exp(zNo(i)).p^{(i)}=\frac{\exp(z^{(i)}_{\texttt{Yes}})}{\exp(z^{(i)}_{\texttt{Yes}})+\exp(z^{(i)}_{\texttt{No}})}. |
|
||||
@@ -0,0 +1 @@
|
||||
| ℒPRM=−∑i=1T[y^(i)⋅logp(i)+(1−y^(i))⋅log(1−p(i))].\mathcal{L}_{\text{PRM}}=-\sum_{i=1}^{T}\left[\hat{y}^{(i)}\cdot\log p^{(i)}+(1-\hat{y}^{(i)})\cdot\log(1-p^{(i)})\right]. |
|
||||
@@ -0,0 +1,16 @@
|
||||
| Model | MM-K12 | OlympiadBench | MathVista | MathVerse | MathVision |
|
||||
| MM-Policy | 33.92 | 15.41 | 62.93 | 42.99 | 21.74 |
|
||||
| +MM-PRM | 42.80 | 24.00 | 67.60 | 46.27 | 27.11 |
|
||||
| +8.88 | +8.59 | +4.67 | +3.28 | +5.37 |
|
||||
| InternVL2.5-8B | 27.01 | 5.23 | 56.43 | 36.26 | 10.04 |
|
||||
| +MM-PRM | 37.80 | 15.33 | 63.50 | 42.56 | 19.41 |
|
||||
| +10.79 | +10.10 | +7.07 | +6.30 | +9.37 |
|
||||
| InternVL2.5-26B | 28.01 | 14.46 | 60.02 | 37.83 | 20.76 |
|
||||
| +MM-PRM | 38.00 | 24.67 | 64.50 | 44.19 | 25.63 |
|
||||
| +9.99 | +10.21 | +4.25 | +6.36 | +4.87 |
|
||||
| InternVL2.5-38B | 40.34 | 29.57 | 68.32 | 47.94 | 29.70 |
|
||||
| +MM-PRM | 52.40 | 32.67 | 71.10 | 52.61 | 32.99 |
|
||||
| +12.06 | +3.10 | +2.78 | +4.67 | +3.29 |
|
||||
| InternVL2.5-78B | 42.24 | 30.98 | 69.48 | 50.18 | 31.50 |
|
||||
| +MM-PRM | 48.80 | 34.67 | 73.20 | 54.47 | 33.26 |
|
||||
| +6.56 | +3.69 | +3.72 | +4.29 | +1.76 |
|
||||
@@ -0,0 +1,3 @@
|
||||
| Labeling Strategy | Min | Average | Max | SumLogPr | SumLogOdds | MeanOdds |
|
||||
| Soft Label | 37.4 | 43.0 | 43.4 | 42.0 | 43.2 | 42.8 |
|
||||
| Hard Label | 36.8 | 34.4 | 35.6 | 36.0 | 33.8 | 37.0 |
|
||||
@@ -0,0 +1,18 @@
|
||||
| Dataset | Sample Count |
|
||||
| R-CoT deng2024r | 201,781 |
|
||||
| MAVIS zhang2024mavis | 29,551 |
|
||||
| multimath-300k peng2024multimath | 224,747 |
|
||||
| Multimodal ArXiv li2024multimodal | 99,772 |
|
||||
| MathV360K shi2024math | 338,721 |
|
||||
| Math-PUMA zhuang2025math | 1,111,939 |
|
||||
| Multi-modal-Self-instruct zhang2024multimodal | 64,807 |
|
||||
| MMPR wang2024mpo | 1,055,796 |
|
||||
| ShareGPT-4o cui2025comprehensive | 57,289 |
|
||||
| table-vqa AgDeTQA | 80,054 |
|
||||
| MathQA amini2019mathqa | 27,923 |
|
||||
| GSM8K cobbe2021training | 7,413 |
|
||||
| DART-Math tong2024dart | 574,305 |
|
||||
| math-gpt-4o-200k pawankrd2024math | 196,032 |
|
||||
| NuminaMath-CoT li2024numinamath | 807,883 |
|
||||
| MathInstruct yue2023mammoth | 257,755 |
|
||||
| Total | 5,135,768 |
|
||||
@@ -0,0 +1 @@
|
||||
| score(rj)=min{p1(j),p2(j),…,pTj(j)}\text{score}(r_{j})=\min\{p_{1}^{(j)},p_{2}^{(j)},\dots,p_{T_{j}}^{(j)}\} |
|
||||
@@ -0,0 +1 @@
|
||||
| score(rj)=max{p1(j),p2(j),…,pTj(j)}\text{score}(r_{j})=\max\{p_{1}^{(j)},p_{2}^{(j)},\dots,p_{T_{j}}^{(j)}\} |
|
||||
@@ -0,0 +1 @@
|
||||
| score(rj)=1Tj∑i=1Tjpi(j)\text{score}(r_{j})=\frac{1}{T_{j}}\sum_{i=1}^{T_{j}}p_{i}^{(j)} |
|
||||
@@ -0,0 +1 @@
|
||||
| score(rj)=∑i=1Tjlogpi(j)=log∏i=1Tjpi(j)\text{score}(r_{j})=\sum_{i=1}^{T_{j}}\log p_{i}^{(j)}=\log\prod_{i=1}^{T_{j}}p_{i}^{(j)} |
|
||||
@@ -0,0 +1 @@
|
||||
| score(rj)=∑i=1Tjlogpi(j)1−pi(j)\text{score}(r_{j})=\sum_{i=1}^{T_{j}}\log\frac{p_{i}^{(j)}}{1-p_{i}^{(j)}} |
|
||||
@@ -0,0 +1 @@
|
||||
| score(rj)=1Tj∑i=1Tjpi(j)1−pi(j)\text{score}(r_{j})=\frac{1}{T_{j}}\sum_{i=1}^{T_{j}}\frac{p_{i}^{(j)}}{1-p_{i}^{(j)}} |
|
||||
@@ -0,0 +1,6 @@
|
||||
| Benchmark | Random | Min | Average | Max | SumLogPr | SumLogOdds | MeanOdds |
|
||||
| MM-K12 | 33.9 | 37.4 | 43.0 | 43.4 | 42.0 | 43.2 | 42.8 |
|
||||
| OlympiadBench | 15.4 | 23.3 | 20.0 | 24.7 | 22.7 | 24.0 | 24.0 |
|
||||
| MathVista | 62.9 | 66.0 | 67.2 | 67.7 | 66.4 | 67.7 | 67.6 |
|
||||
| MathVerse | 43.0 | 46.0 | 48.0 | 46.1 | 47.3 | 48.0 | 46.3 |
|
||||
| MathVision | 21.7 | 25.8 | 26.7 | 26.9 | 25.4 | 26.3 | 27.1 |
|
||||
@@ -0,0 +1,6 @@
|
||||
| Benchmark | Random | Min | Average | Max | SumLogPr | SumLogOdds | MeanOdds |
|
||||
| MM-K12 | 27.0 | 34.8 | 36.0 | 35.8 | 31.2 | 34.2 | 37.8 |
|
||||
| OlympiadBench | 5.2 | 2.7 | 15.3 | 16.7 | 2.0 | 1.3 | 15.3 |
|
||||
| MathVista | 56.4 | 63.4 | 62.8 | 62.6 | 61.0 | 62.0 | 63.5 |
|
||||
| MathVerse | 36.3 | 41.8 | 42.8 | 41.1 | 42.0 | 42.8 | 42.6 |
|
||||
| MathVision | 10.0 | 6.1 | 18.4 | 20.6 | 3.8 | 4.8 | 19.4 |
|
||||
@@ -0,0 +1,6 @@
|
||||
| Benchmark | Random | Min | Average | Max | SumLogPr | SumLogOdds | MeanOdds |
|
||||
| MM-K12 | 28.0 | 38.0 | 37.2 | 35.4 | 32.0 | 33.0 | 38.0 |
|
||||
| OlympiadBench | 14.5 | 16.0 | 23.3 | 22.0 | 16.7 | 16.7 | 24.7 |
|
||||
| MathVista | 60.0 | 63.9 | 64.6 | 64.4 | 62.8 | 64.2 | 64.5 |
|
||||
| MathVerse | 37.8 | 42.0 | 44.2 | 44.3 | 40.1 | 41.5 | 44.2 |
|
||||
| MathVision | 20.8 | 23.3 | 24.1 | 24.9 | 20.6 | 22.0 | 25.6 |
|
||||
@@ -0,0 +1,6 @@
|
||||
| Benchmark | Random | Min | Average | Max | SumLogPr | SumLogOdds | MeanOdds |
|
||||
| MM-K12 | 40.3 | 46.8 | 51.0 | 49.8 | 42.8 | 46.6 | 52.4 |
|
||||
| OlympiadBench | 29.6 | 29.3 | 34.0 | 34.7 | 28.0 | 30.0 | 32.7 |
|
||||
| MathVista | 68.3 | 70.5 | 71.1 | 70.5 | 69.2 | 70.1 | 71.1 |
|
||||
| MathVerse | 47.9 | 50.4 | 51.9 | 51.1 | 51.1 | 52.7 | 52.6 |
|
||||
| MathVision | 29.7 | 29.1 | 32.6 | 33.6 | 30.6 | 31.3 | 33.0 |
|
||||
@@ -0,0 +1,6 @@
|
||||
| Benchmark | Random | Min | Average | Max | SumLogPr | SumLogOdds | MeanOdds |
|
||||
| MM-K12 | 42.2 | 46.2 | 47.2 | 47.2 | 45.0 | 46.8 | 48.8 |
|
||||
| OlympiadBench | 31.0 | 33.3 | 35.3 | 34.7 | 33.3 | 34.7 | 34.7 |
|
||||
| MathVista | 69.5 | 71.7 | 73.4 | 72.4 | 73.9 | 73.6 | 73.2 |
|
||||
| MathVerse | 50.2 | 53.4 | 55.0 | 54.2 | 54.3 | 55.1 | 54.5 |
|
||||
| MathVision | 31.5 | 30.1 | 33.5 | 33.6 | 31.8 | 32.4 | 33.3 |
|
||||
@@ -0,0 +1,138 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independent verifier for the MM-PRM archive.
|
||||
|
||||
Recomputes every hash from the files on disk and compares against manifest.json
|
||||
(the manifest is the CLAIM; the files are the EVIDENCE). Also performs a
|
||||
long-context grounding test: every chunk's needle must occur exactly once in the
|
||||
full paper text and land inside that chunk's recorded [start, end) range.
|
||||
|
||||
Exit 0 = archive is self-consistent and correctly grounded.
|
||||
Exit 1 = at least one check failed.
|
||||
"""
|
||||
import hashlib, json, os, sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
|
||||
def sha256_file(p):
|
||||
h = hashlib.sha256()
|
||||
with open(p, "rb") as f:
|
||||
for b in iter(lambda: f.read(65536), b""):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def main():
|
||||
fails = []
|
||||
|
||||
def check(cond, msg):
|
||||
print(("PASS: " if cond else "FAIL: ") + msg)
|
||||
if not cond:
|
||||
fails.append(msg)
|
||||
|
||||
mp = os.path.join(HERE, "manifest.json")
|
||||
check(os.path.exists(mp), "manifest.json exists")
|
||||
if not os.path.exists(mp):
|
||||
print("VERDICT: FAIL")
|
||||
return 1
|
||||
m = json.load(open(mp, encoding="utf-8"))
|
||||
|
||||
# 1. source provenance
|
||||
sp = os.path.join(HERE, "source.html")
|
||||
check(os.path.exists(sp), "source.html exists (raw provenance)")
|
||||
if os.path.exists(sp):
|
||||
check(sha256_file(sp) == m["source_sha256"], "source.html sha256 == manifest.source_sha256")
|
||||
|
||||
# 2. paper text
|
||||
pp = os.path.join(HERE, "paper.txt")
|
||||
check(os.path.exists(pp), "paper.txt exists")
|
||||
paper = open(pp, encoding="utf-8").read()
|
||||
check(sha256_file(pp) == m["paper_txt_sha256"], "paper.txt sha256 == manifest.paper_txt_sha256")
|
||||
check(len(paper) == m["paper_chars"], "paper.txt length == manifest.paper_chars (%d)" % m["paper_chars"])
|
||||
check(m["title"] in paper, "paper title present in paper.txt")
|
||||
|
||||
# 3. every manifest heading is actually in the extracted text
|
||||
miss = [h for h in m["headings"] if h not in paper]
|
||||
check(not miss, "all %d manifest headings present in paper.txt" % len(m["headings"]))
|
||||
if miss:
|
||||
print(" missing:", miss[:5])
|
||||
|
||||
# 4. chunks: exist, hash-match, exactly tile paper.txt
|
||||
chunks = sorted(m["chunks"], key=lambda c: c["index"])
|
||||
check(len(chunks) == m["num_chunks"], "chunk count == manifest.num_chunks (%d)" % m["num_chunks"])
|
||||
step = m["chunk_size"] - m["chunk_overlap"]
|
||||
tiling_ok = True
|
||||
prev_start = None
|
||||
for c in chunks:
|
||||
fp = os.path.join(HERE, c["file"])
|
||||
if not os.path.exists(fp):
|
||||
check(False, "chunk %03d file exists" % c["index"])
|
||||
tiling_ok = False
|
||||
continue
|
||||
data = open(fp, encoding="utf-8").read()
|
||||
if sha256_file(fp) != c["sha256"]:
|
||||
check(False, "chunk %03d sha256 matches" % c["index"])
|
||||
tiling_ok = False
|
||||
if data != paper[c["start"]:c["end"]]:
|
||||
check(False, "chunk %03d text == paper.txt[start:end]" % c["index"])
|
||||
tiling_ok = False
|
||||
if not (0 <= c["start"] < c["end"] <= len(paper)):
|
||||
tiling_ok = False
|
||||
if prev_start is None:
|
||||
if c["start"] != 0:
|
||||
tiling_ok = False
|
||||
elif c["start"] != prev_start + step:
|
||||
tiling_ok = False
|
||||
prev_start = c["start"]
|
||||
check(tiling_ok, "chunks tile paper.txt contiguously (step=%d, overlap=%d)" % (step, m["chunk_overlap"]))
|
||||
check(chunks[-1]["end"] == len(paper), "last chunk ends exactly at paper.txt end")
|
||||
|
||||
# 5. long-context grounding: needle unique in full text, located in its chunk
|
||||
ground_ok = True
|
||||
for c in chunks:
|
||||
n = c.get("needle")
|
||||
if not n:
|
||||
ground_ok = False
|
||||
print(" chunk %03d has no needle" % c["index"])
|
||||
continue
|
||||
cnt = paper.count(n)
|
||||
if cnt != 1:
|
||||
ground_ok = False
|
||||
print(" chunk %03d needle not unique in paper.txt (count=%d)" % (c["index"], cnt))
|
||||
continue
|
||||
pos = paper.find(n)
|
||||
if not (c["start"] <= pos < c["end"]):
|
||||
ground_ok = False
|
||||
print(" chunk %03d needle at %d outside [%d,%d)" % (c["index"], pos, c["start"], c["end"]))
|
||||
check(ground_ok, "all %d needles unique and located in their recorded chunk (grounding)" % len(chunks))
|
||||
|
||||
# 6. tables
|
||||
t_ok = len(m["tables"]) == m["num_tables"]
|
||||
for t in m["tables"]:
|
||||
fp = os.path.join(HERE, t["file"])
|
||||
if not os.path.exists(fp) or sha256_file(fp) != t["sha256"] or os.path.getsize(fp) == 0:
|
||||
t_ok = False
|
||||
print(" table %02d bad" % t["index"])
|
||||
check(t_ok, "all %d tables present, hashed, non-empty" % len(m["tables"]))
|
||||
|
||||
# 7. figures
|
||||
f_ok = len(m["figures"]) == m["num_figures"]
|
||||
for f in m["figures"]:
|
||||
fp = os.path.join(HERE, f["file"])
|
||||
if (not os.path.exists(fp) or sha256_file(fp) != f["sha256"]
|
||||
or not f.get("fetch_ok") or os.path.getsize(fp) == 0):
|
||||
f_ok = False
|
||||
print(" figure %02d bad" % f["index"])
|
||||
check(f_ok, "all %d figures present, hashed, non-empty, fetch_ok" % len(m["figures"]))
|
||||
|
||||
print()
|
||||
if fails:
|
||||
print("VERDICT: FAIL (%d failed checks)" % len(fails))
|
||||
return 1
|
||||
print("VERDICT: PASS - archive self-consistent: %d chunks, %d tables, %d figures, %d chars"
|
||||
% (len(chunks), len(m["tables"]), len(m["figures"]), len(paper)))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -9,13 +9,16 @@
|
||||
"ocr_psm": "11",
|
||||
"ocr_word_count_raw": 69,
|
||||
"ocr_word_count_joined": 67,
|
||||
"ocr_candidate_count": 110,
|
||||
"ocr_vertical_stack_count": 21,
|
||||
"ocr_group_count": 40,
|
||||
"labels_graded": 13,
|
||||
"ocr_exact": 7,
|
||||
"ocr_boundary": 6,
|
||||
"ocr_exact": 6,
|
||||
"ocr_boundary": 7,
|
||||
"ocr_absent": 0,
|
||||
"faults": [
|
||||
{
|
||||
"label": "Inputs",
|
||||
"ocr_status": "OCR_BOUNDARY"
|
||||
},
|
||||
{
|
||||
"label": "Input Embedding",
|
||||
"ocr_status": "OCR_BOUNDARY"
|
||||
@@ -44,18 +47,12 @@
|
||||
"rows": [
|
||||
{
|
||||
"label": "Inputs",
|
||||
"ocr_status": "OCR_EXACT",
|
||||
"ocr_status": "OCR_BOUNDARY",
|
||||
"present_in_text": false,
|
||||
"tokens": [
|
||||
"inputs"
|
||||
],
|
||||
"ocr_run": [
|
||||
{
|
||||
"text": "Inputs",
|
||||
"x": 412,
|
||||
"y": 2102
|
||||
}
|
||||
]
|
||||
"ocr_run": []
|
||||
},
|
||||
{
|
||||
"label": "Input Embedding",
|
||||
@@ -87,8 +84,8 @@
|
||||
"ocr_run": [
|
||||
{
|
||||
"text": "Nx",
|
||||
"x": 47,
|
||||
"y": 1234
|
||||
"x": 1387,
|
||||
"y": 1045
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -118,6 +115,11 @@
|
||||
"norm"
|
||||
],
|
||||
"ocr_run": [
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 511
|
||||
},
|
||||
{
|
||||
"text": "}<s",
|
||||
"x": 1199,
|
||||
@@ -202,11 +204,6 @@
|
||||
"linear"
|
||||
],
|
||||
"ocr_run": [
|
||||
{
|
||||
"text": "J",
|
||||
"x": 1197,
|
||||
"y": 355
|
||||
},
|
||||
{
|
||||
"text": "Linear",
|
||||
"x": 964,
|
||||
@@ -222,6 +219,11 @@
|
||||
"softmax"
|
||||
],
|
||||
"ocr_run": [
|
||||
{
|
||||
"text": "l",
|
||||
"x": 845,
|
||||
"y": 215
|
||||
},
|
||||
{
|
||||
"text": "|",
|
||||
"x": 1198,
|
||||
@@ -245,69 +247,26 @@
|
||||
"ocr_run": []
|
||||
}
|
||||
],
|
||||
"ocr_vertical_stacks": [
|
||||
"ocr_groups": [
|
||||
[
|
||||
{
|
||||
"text": "|",
|
||||
"x": 1198,
|
||||
"y": 215
|
||||
},
|
||||
{
|
||||
"text": "J",
|
||||
"x": 1197,
|
||||
"y": 355
|
||||
},
|
||||
{
|
||||
"text": "}<s",
|
||||
"x": 1199,
|
||||
"y": 511
|
||||
},
|
||||
{
|
||||
"text": "eA",
|
||||
"x": 1196,
|
||||
"y": 825
|
||||
},
|
||||
{
|
||||
"text": "Jen",
|
||||
"x": 1198,
|
||||
"y": 1191
|
||||
},
|
||||
{
|
||||
"text": "=i",
|
||||
"x": 884,
|
||||
"y": 1483
|
||||
"text": "Output",
|
||||
"x": 939,
|
||||
"y": 8
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "J",
|
||||
"text": "Probabilities",
|
||||
"x": 873,
|
||||
"y": 83
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "t",
|
||||
"x": 1017,
|
||||
"y": 433
|
||||
},
|
||||
{
|
||||
"text": "&",
|
||||
"x": 998,
|
||||
"y": 521
|
||||
},
|
||||
{
|
||||
"text": "Feed",
|
||||
"x": 976,
|
||||
"y": 614
|
||||
},
|
||||
{
|
||||
"text": "Forward",
|
||||
"x": 939,
|
||||
"y": 678
|
||||
},
|
||||
{
|
||||
"text": "Multi-Head",
|
||||
"x": 908,
|
||||
"y": 928
|
||||
},
|
||||
{
|
||||
"text": "_}",
|
||||
"x": 832,
|
||||
"y": 1058
|
||||
"y": 140
|
||||
}
|
||||
],
|
||||
[
|
||||
@@ -316,230 +275,29 @@
|
||||
"x": 845,
|
||||
"y": 215
|
||||
},
|
||||
{
|
||||
"text": "(",
|
||||
"x": 845,
|
||||
"y": 355
|
||||
},
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 511
|
||||
},
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 825
|
||||
},
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 1191
|
||||
},
|
||||
{
|
||||
"text": "N",
|
||||
"x": 812,
|
||||
"y": 1575
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "t",
|
||||
"x": 1017,
|
||||
"y": 140
|
||||
"x": 1198,
|
||||
"y": 215
|
||||
},
|
||||
{
|
||||
"text": "Softmax",
|
||||
"x": 936,
|
||||
"y": 224
|
||||
},
|
||||
{
|
||||
"text": "Attention",
|
||||
"x": 927,
|
||||
"y": 992
|
||||
},
|
||||
{
|
||||
"text": "Multi-Head",
|
||||
"x": 908,
|
||||
"y": 1355
|
||||
},
|
||||
{
|
||||
"text": "Embedding",
|
||||
"x": 902,
|
||||
"y": 1899
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "&",
|
||||
"x": 998,
|
||||
"y": 835
|
||||
},
|
||||
{
|
||||
"text": "j}",
|
||||
"x": 990,
|
||||
"y": 1054
|
||||
},
|
||||
{
|
||||
"text": "Masked",
|
||||
"x": 943,
|
||||
"y": 1291
|
||||
},
|
||||
{
|
||||
"text": "Attention",
|
||||
"x": 935,
|
||||
"y": 1418
|
||||
},
|
||||
{
|
||||
"text": "Outputs",
|
||||
"x": 924,
|
||||
"y": 2101
|
||||
"text": "(",
|
||||
"x": 845,
|
||||
"y": 355
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 887,
|
||||
"y": 521
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 887,
|
||||
"y": 835
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 887,
|
||||
"y": 1201
|
||||
},
|
||||
{
|
||||
"text": "(shifted",
|
||||
"x": 862,
|
||||
"y": 2175
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Norm",
|
||||
"x": 1047,
|
||||
"y": 521
|
||||
},
|
||||
{
|
||||
"text": "Norm",
|
||||
"x": 1047,
|
||||
"y": 835
|
||||
},
|
||||
{
|
||||
"text": "Norm",
|
||||
"x": 1047,
|
||||
"y": 1201
|
||||
},
|
||||
{
|
||||
"text": "Vas:",
|
||||
"x": 1000,
|
||||
"y": 1573
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Feed",
|
||||
"x": 436,
|
||||
"y": 982
|
||||
},
|
||||
{
|
||||
"text": "Forward",
|
||||
"x": 399,
|
||||
"y": 1045
|
||||
},
|
||||
{
|
||||
"text": "Multi-Head",
|
||||
"x": 369,
|
||||
"y": 1350
|
||||
},
|
||||
{
|
||||
"text": "A__f",
|
||||
"x": 345,
|
||||
"y": 1483
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "yy,",
|
||||
"x": 1269,
|
||||
"y": 770
|
||||
},
|
||||
{
|
||||
"text": "Positional",
|
||||
"x": 1269,
|
||||
"y": 1672
|
||||
},
|
||||
{
|
||||
"text": "Encoding",
|
||||
"x": 1269,
|
||||
"y": 1747
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 347,
|
||||
"y": 892
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 347,
|
||||
"y": 1261
|
||||
},
|
||||
{
|
||||
"text": "OF",
|
||||
"x": 288,
|
||||
"y": 1665
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "&Norm",
|
||||
"x": 458,
|
||||
"y": 892
|
||||
},
|
||||
{
|
||||
"text": "&Norm",
|
||||
"x": 458,
|
||||
"y": 1261
|
||||
},
|
||||
{
|
||||
"text": "Embedding",
|
||||
"x": 362,
|
||||
"y": 1899
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Nx",
|
||||
"x": 47,
|
||||
"y": 1234
|
||||
},
|
||||
{
|
||||
"text": "Positional",
|
||||
"x": 5,
|
||||
"y": 1661
|
||||
},
|
||||
{
|
||||
"text": "Encoding",
|
||||
"x": 5,
|
||||
"y": 1736
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Output",
|
||||
"x": 939,
|
||||
"y": 8
|
||||
},
|
||||
{
|
||||
"text": "Probabilities",
|
||||
"x": 873,
|
||||
"y": 83
|
||||
"text": "J",
|
||||
"x": 1197,
|
||||
"y": 355
|
||||
}
|
||||
],
|
||||
[
|
||||
@@ -547,11 +305,88 @@
|
||||
"text": "Linear",
|
||||
"x": 964,
|
||||
"y": 365
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "J",
|
||||
"x": 1017,
|
||||
"y": 433
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 511
|
||||
},
|
||||
{
|
||||
"text": "Output",
|
||||
"x": 952,
|
||||
"y": 1834
|
||||
"text": "}<s",
|
||||
"x": 1199,
|
||||
"y": 511
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 887,
|
||||
"y": 521
|
||||
},
|
||||
{
|
||||
"text": "&",
|
||||
"x": 998,
|
||||
"y": 521
|
||||
},
|
||||
{
|
||||
"text": "Norm",
|
||||
"x": 1047,
|
||||
"y": 521
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Feed",
|
||||
"x": 976,
|
||||
"y": 614
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Forward",
|
||||
"x": 939,
|
||||
"y": 678
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "yy,",
|
||||
"x": 1269,
|
||||
"y": 770
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 825
|
||||
},
|
||||
{
|
||||
"text": "eA",
|
||||
"x": 1196,
|
||||
"y": 825
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 887,
|
||||
"y": 835
|
||||
},
|
||||
{
|
||||
"text": "&",
|
||||
"x": 998,
|
||||
"y": 835
|
||||
},
|
||||
{
|
||||
"text": "Norm",
|
||||
"x": 1047,
|
||||
"y": 835
|
||||
}
|
||||
],
|
||||
[
|
||||
@@ -561,45 +396,42 @@
|
||||
"y": 882
|
||||
},
|
||||
{
|
||||
"text": "|",
|
||||
"x": 658,
|
||||
"y": 1251
|
||||
"text": "Add",
|
||||
"x": 347,
|
||||
"y": 892
|
||||
},
|
||||
{
|
||||
"text": "&Norm",
|
||||
"x": 458,
|
||||
"y": 892
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Sf",
|
||||
"x": 1111,
|
||||
"y": 1054
|
||||
},
|
||||
{
|
||||
"text": "right)",
|
||||
"x": 1071,
|
||||
"y": 2175
|
||||
"text": "Multi-Head",
|
||||
"x": 908,
|
||||
"y": 928
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "L",
|
||||
"x": 477,
|
||||
"y": 1127
|
||||
},
|
||||
"text": "Feed",
|
||||
"x": 436,
|
||||
"y": 982
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Attention",
|
||||
"x": 387,
|
||||
"y": 1414
|
||||
"x": 927,
|
||||
"y": 992
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Input",
|
||||
"x": 436,
|
||||
"y": 1835
|
||||
},
|
||||
{
|
||||
"text": "Inputs",
|
||||
"x": 412,
|
||||
"y": 2102
|
||||
"text": "Forward",
|
||||
"x": 399,
|
||||
"y": 1045
|
||||
}
|
||||
],
|
||||
[
|
||||
@@ -610,18 +442,226 @@
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "j}",
|
||||
"x": 990,
|
||||
"y": 1054
|
||||
},
|
||||
{
|
||||
"text": "Sf",
|
||||
"x": 1111,
|
||||
"y": 1054
|
||||
},
|
||||
{
|
||||
"text": "_}",
|
||||
"x": 832,
|
||||
"y": 1058
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "L",
|
||||
"x": 477,
|
||||
"y": 1127
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "|",
|
||||
"x": 845,
|
||||
"y": 1191
|
||||
},
|
||||
{
|
||||
"text": "Jen",
|
||||
"x": 1198,
|
||||
"y": 1191
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 887,
|
||||
"y": 1201
|
||||
},
|
||||
{
|
||||
"text": "&",
|
||||
"x": 998,
|
||||
"y": 1201
|
||||
},
|
||||
{
|
||||
"text": "Norm",
|
||||
"x": 1047,
|
||||
"y": 1201
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Nx",
|
||||
"x": 47,
|
||||
"y": 1234
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "|",
|
||||
"x": 658,
|
||||
"y": 1251
|
||||
},
|
||||
{
|
||||
"text": "Add",
|
||||
"x": 347,
|
||||
"y": 1261
|
||||
},
|
||||
{
|
||||
"text": "&Norm",
|
||||
"x": 458,
|
||||
"y": 1261
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Masked",
|
||||
"x": 943,
|
||||
"y": 1291
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Multi-Head",
|
||||
"x": 369,
|
||||
"y": 1350
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Multi-Head",
|
||||
"x": 908,
|
||||
"y": 1355
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Attention",
|
||||
"x": 387,
|
||||
"y": 1414
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Attention",
|
||||
"x": 935,
|
||||
"y": 1418
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "A__f",
|
||||
"x": 345,
|
||||
"y": 1483
|
||||
},
|
||||
{
|
||||
"text": "=i",
|
||||
"x": 884,
|
||||
"y": 1483
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Vas:",
|
||||
"x": 1000,
|
||||
"y": 1573
|
||||
},
|
||||
{
|
||||
"text": "\\",
|
||||
"x": 179,
|
||||
"y": 1581
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "N",
|
||||
"x": 812,
|
||||
"y": 1575
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Positional",
|
||||
"x": 5,
|
||||
"y": 1661
|
||||
},
|
||||
{
|
||||
"text": "OF",
|
||||
"x": 288,
|
||||
"y": 1665
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Positional",
|
||||
"x": 1269,
|
||||
"y": 1672
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Encoding",
|
||||
"x": 5,
|
||||
"y": 1736
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Encoding",
|
||||
"x": 1269,
|
||||
"y": 1747
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Output",
|
||||
"x": 952,
|
||||
"y": 1834
|
||||
},
|
||||
{
|
||||
"text": "Input",
|
||||
"x": 436,
|
||||
"y": 1835
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Embedding",
|
||||
"x": 362,
|
||||
"y": 1899
|
||||
},
|
||||
{
|
||||
"text": "Embedding",
|
||||
"x": 902,
|
||||
"y": 1899
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "Outputs",
|
||||
"x": 924,
|
||||
"y": 2101
|
||||
},
|
||||
{
|
||||
"text": "Inputs",
|
||||
"x": 412,
|
||||
"y": 2102
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"text": "(shifted",
|
||||
"x": 862,
|
||||
"y": 2175
|
||||
},
|
||||
{
|
||||
"text": "right)",
|
||||
"x": 1071,
|
||||
"y": 2175
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user