OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Models

Large vision-language models (LVLMs) have made substantial advances in tackling Olympiad-level reasoning tasks. However, existing benchmarks primarily emphasize single-image analysis, failing to test the ability to synthesize information across multiple images—a capability essential for real-world agentic systems handling dashboards, documents, or experimental setups.

This gap leaves builders uncertain about model performance in distributed visual contexts. OMIBench addresses this by curating Olympiad problems from biology, chemistry, mathematics, and physics where critical evidence is spread across multiple images.

What the Paper Does

OMIBench provides manually annotated rationales and dual evaluation protocols: exact matching for precision and semantic matching for nuanced understanding. The benchmark challenges models on cross-image integration, revealing hierarchical weaknesses.

Key results: Even leading LVLMs like Gemini-3-Pro score only ~50%. This quantifies the drop-off in multi-image scenarios compared to single-image baselines, with consistent gaps across domains.

Why It Matters for Builders

Multimodal agents in production often process sequences of screenshots, multi-panel figures, or UI elements. Single-image evals overestimate capabilities; OMIBench exposes failures in visual evidence aggregation, critical for reliable deployment in research tools, diagnostics, or creative apps. It enables targeted improvements in attention, retrieval, or CoT for visuals.

Builder Takeaway

Grab OMIBench from Hugging Face (https://huggingface.co/datasets/LightChen2333/OMIBench) or GitHub (https://github.com/LightChen233/OMIBench) and eval your VLMs—expect drops below 50% on top models. Prioritize multi-image CoT prompting or fine-tuning to close the gap.

Source: Qiguang Chen, Chengyu Luan, Jiajun Wu, Qiming Yu, Yi Yang, Yizhuo Li, Jingqi Tong, Xiachong Feng, Libo Qin, Wanxiang Che — ArXiv cs.CV, April 2026