Judge Anything: MLLM as a Judge Across Any Modality

摘要

Evaluating generative foundation models on open-ended multimodal understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g., images, audio, video) poses significant challenges due to the complexity of cross-modal interactions. To this end, the concept of utilizing Multimodal LLMs (MLLMs) as automated judges has emerged, yielding encouraging results in assessing vision-language understanding tasks. Building upon this, this paper extends the MLLM-as-a-Judge paradigm to unify evaluation across modalities. We introduce TaskAnything and JudgeAnything, to respectively evaluate the overall performance and judging capabilities of MLLMs across any-to-any modality tasks. Specifically, TaskAnything evaluates the MMU and MMG capabilities across 15 any-to-any modality categories, employing 1,500 queries curated from well-established benchmarks. Furthermore, JudgeAnything evaluates the judging capabilities of 8 advanced models (e.g., Gemini-1.5-Pro and Qwen2.5-Omni series) from the perspectives of Pair Comparison and Score Evaluation, providing a standardized testbed that incorporates human judgments and detailed rubrics. Our extensive experiments reveal that while these MLLMs show promise in assessing MMU (i.e., achieving an average of 66.55% under the Pair Comparison setting and 42.79% under the Score Evaluation setting), they encounter significant challenges with MMG tasks (i.e., averaging only 53.37% under the Pair Comparison setting and 30.05% under the Score Evaluation setting), exposing cross-modality biases and hallucination issues. To address this, we present OmniArena, an automated platform for evaluating omni-models and multimodal reward models. Our work highlights the need for fairer evaluation protocols and stronger alignment with human preferences. The source code and dataset are publicly available at: https://urrealhero.github.io/judgeanythingweb/.