Evaluating the effectiveness of a security flaws prevention tool

Context: Securing code is crucial for all software stakeholders. Nevertheless, state-of-the-art tools are imperfect and tend to miss critical errors, resulting in zero-day vulnerabilities. Thus, there is a need for alternatives to mitigate such issues. Objective: We aim to facilitate an effective identification mechanism of security flaws in the early stages of development. Method: Following our analysis of the root causes of vulnerabilities and examining existing code analyzers, we devise a new Rule-Based Security Flaws Prevention (RbSFP) tool. The tool is based on a set of allow-list rules and consists of the following stages: (1) AST creation based on the source code and marking critical code areas; (2) Context-based code analysis that further validates the code; (3) Results’ normalization to suggest alerts and warnings. To evaluate the RbSFP tool, we utilized two complementary evaluations. The first refers to the tool's ability to detect security flaws compared to competing tools by executing them on open-source projects. The second refers to evaluating the tool's usability and efficiency via a controlled experiment. Results: We found that the outcomes were of better quality when using the RbSFP tool, and the differences were statistically significant. Thus, utilizing the new approach and tool has a significant impact as it can eliminate root causes for security flaws at the early stages of development. Conclusion: Using an allow-list-based approach can reduce security flaws in the code. However, further analysis and evaluation are needed to provide a more comprehensive solution.