Journal article
Beyond Fidelity: Explaining Vulnerability Localization of Learning-Based Detectors
ACM transactions on software engineering and methodology, Vol.33(5), pp.1-33
06/2024
Metrics
43 Record Views
Abstract
Vulnerability detectors based on deep learning (DL) models have proven their effectiveness in recent years. However, the shroud of opacity surrounding the decision-making process of these detectors makes it difficult for security analysts to comprehend. To address this, various explanation approaches have been proposed to explain the predictions by highlighting important features, which have been demonstrated effective in domains such as computer vision and natural language processing. Unfortunately, there is still a lack of in-depth evaluation of vulnerability-critical features, such as fine-grained vulnerability-related code lines, learned and understood by these explanation approaches. In this study, we first evaluate the performance of ten explanation approaches for vulnerability detectors based on graph and sequence representations, measured by two quantitative metrics including fidelity and vulnerability line coverage rate. Our results show that fidelity alone is insufficent for evaluating these approaches, as fidelity incurs significant fluctuations across different datasets and detectors. We subsequently check the precision of the vulnerability-related code lines reported by the explanation approaches, and find poor accuracy in this task among all of them. This can be attributed to the inefficiency of explainers in selecting important features and the presence of irrelevant artifacts learned by DL-based detectors.
Details
- Title
- Beyond Fidelity: Explaining Vulnerability Localization of Learning-Based Detectors
- Creators
- Baijun Cheng - Peking UniversityShengming Zhao - University of AlbertaKailong Wang - Huazhong University of Science and TechnologyMeizhen Wang - Huazhong University of Science and TechnologyGuangdong Bai - The University of QueenslandRuitao Feng - UNSW SydneyYao Guo - Peking UniversityLei Ma - University of AlbertaHaoyu Wang - Huazhong University of Science and Technology
- Publication Details
- ACM transactions on software engineering and methodology, Vol.33(5), pp.1-33
- Publisher
- Association for Computing Machinery
- Identifiers
- 991013214583202368
- Academic Unit
- Information Technology; Faculty of Science and Engineering
- Language
- English
- Resource Type
- Journal article