Files and links (2)
Preprint (Author's original)CC BY V4.0, Open Access
Preprint (Author's original)CC BY V4.0, Open
Preprint
AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
arXiv (Cornell University)
Cornell University
27/01/2026
Metrics
1 Record Views
Abstract
High-quality evaluation benchmarks are pivotal for deploying Large Language Models (LLMs) in Automated Code Review (ACR). However, existing benchmarks suffer from two critical limitations: first, the lack of multi-language support in repository-level contexts, which restricts the generalizability of evaluation results; second, the reliance on noisy, incomplete ground truth derived from raw Pull Request (PR) comments, which constrains the scope of issue detection. To address these challenges, we introduce AACR-Bench a comprehensive benchmark that provides full cross-file context across multiple programming languages. Unlike traditional datasets, AACR-Bench employs an "AI-assisted, Expert-verified" annotation pipeline to uncover latent defects often overlooked in original PRs, resulting in a 285\% increase in defect coverage. Extensive evaluations of mainstream LLMs on AACR-Bench reveal that previous assessments may have either misjudged or only partially captured model capabilities due to data limitations. Our work establishes a more rigorous standard for ACR evaluation and offers new insights on LLM based ACR, i.e., the granularity/level of context and the choice of retrieval methods significantly impact ACR performance, and this influence varies depending on the LLM, programming language, and the LLM usage paradigm e.g., whether an Agent architecture is employed. The code, data, and other artifacts of our evaluation set are available at https://github.com/alibaba/aacr-bench .
Details
- Title
- AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
- Creators
- Lei Zhang - Nanjing UniversityYongda Yu - Nanjing UniversityMinghui Yu - Nanjing UniversityXinxin Guo - Nanjing UniversityZhengqi Zhuang - Nanjing UniversityGuoping Rong - Nanjing UniversityDong Shao - Nanjing UniversityHaifeng Shen - Southern Cross UniversityHongyu Kuang - Nanjing UniversityZhengfeng Li - Alibaba Group (China)Boge Wang - Alibaba Group (China)Guoan Zhang - Alibaba Group (China)Bangyu Xiang - Alibaba Group (China)Xiaobing Xu - Alibaba Group (China)
- Publication Details
- arXiv (Cornell University)
- Publisher
- Cornell University
- Identifiers
- 991013345418902368
- Academic Unit
- Faculty of Science and Engineering
- Language
- English
- Resource Type
- Preprint