Logo image
Back
Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java
Conference proceeding   Peer reviewed

Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java

Kaixuan Li, Sen Chen, Lingling Fan, Ruitao Feng, Han Liu, Chengwei Liu, Yang Liu and Yixiang Chen
ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp.921-933
ACM Conferences
ESEC/FSE '23: 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, USA, 03/12/2023–09/12/2023)
30/11/2023
DOI: https://doi.org/10.1145/3611643.3616262

Metrics

28 Record Views
20 Times Cited - Web of Science
Article has an altmetric score of 1

See more details

40 readers on Mendeley

Abstract

Benchmarks Empirical study Static application security testing
Static application security testing (SAST) takes a significant role in the software development life cycle (SDLC). However, it is challenging to comprehensively evaluate the effectiveness of SAST tools to determine which is the better one for detecting vulnerabilities. In this paper, based on well-defined criteria, we first selected seven free or open-source SAST tools from 161 existing tools for further evaluation. Owing to the synthetic and newly-constructed real-world benchmarks, we evaluated and compared these SAST tools from different and comprehensive perspectives such as effectiveness, consistency, and performance. While SAST tools perform well on synthetic benchmarks, our results indicate that only 12.7% of real-world vulnerabilities can be detected by the selected tools. Even combining the detection capability of all tools, most vulnerabilities (70.9%) remain undetected, especially those beyond resource control and insufficiently neutralized input/output vulnerabilities. The fact is that although they have already built the corresponding detecting rules and integrated them into their capabilities, the detection result still did not meet the expectations. All useful findings unveiled in our comprehensive study indeed help to provide guidance on tool development, improvement, evaluation, and selection for developers, researchers, and potential users.

Details

Logo image