MJ-Video: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation

Haibo Tong*,1      Zhaoyang Wang*,1      Zhaorun Chen2      Haonian Ji1      Shi Qiu1      Siwei Han1      Kexin Geng1      Zhongkai Xue3     
Yiyang Zhou1      Peng Xia1      Mingyu Ding1      Rafael Rafailov4      Chelsea Finn4      Huaxiu Yao1

1UNC-Chapel Hill      2University of Chicago      3University of Oxford      4Stanford University     
MJ-Bench Team
*Equal Contribution.
Paper Dataset Code Demo
Main Figure

Abstract

Recent advancements in video generation have significantly improved the ability to synthesize videos from text instructions. However, existing models still struggle with key challenges such as instruction misalignment, content hallucination, safety concerns, and bias. Addressing these limitations, we introduce MJ-BENCH-VIDEO, a large-scale video preference benchmark designed to evaluate video generation across five critical aspects: Alignment, Safety, Fineness, Coherence & Consistency, and Bias & Fairness. This benchmark incorporates 28 fine-grained criteria to provide a comprehensive evaluation of video preference. Building upon this dataset, we propose MJ-VIDEO, a Mixture-of-Experts (MoE)-based video reward model designed to deliver fine-grained reward. MJ-VIDEO can dynamically select relevant experts to accurately judge the preference based on the input text-video pair. This architecture enables more precise and adaptable preference judgments. Through extensive benchmarking on MJ-BENCH-VIDEO, we analyze the limitations of existing video reward models and demonstrate the superior performance of MJ-VIDEO in video preference assessment, achieving 17.58% and 15.87% improvements in overall and fine-grained preference judgments, respectively. Additionally, introducing MJ-VIDEO for preference tuning in video generation enhances the alignment performance.

Video Prefered by MJ-VIDEO and VideoScore

MJ-VIDEO Vs VideoScore for Video Alignment

BibTeX

@misc{tong2025mjvideofinegrainedbenchmarkingrewarding,
      title={MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation}, 
      author={Haibo Tong and Zhaoyang Wang and Zhaorun Chen and Haonian Ji and Shi Qiu and Siwei Han and Kexin Geng and Zhongkai Xue and Yiyang Zhou and Peng Xia and Mingyu Ding and Rafael Rafailov and Chelsea Finn and Huaxiu Yao},
      year={2025},
      eprint={2502.01719},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.01719}, 
}

Contact Us

If you have any inquiries about MJ-VIDEO, feel free to reach out to us at mjbenchofficial@gmail.com or raise an issue on Github.