Motivation: Software reliability and security are critically important. Software bugs and vulnerabilites greatly hurt software reliability.
A fundamental challenge of detecting or preventing software bugs and vulnerabilites is to know programmers' intentions, formally called specifications. If we know the specification of a program (e.g., where a lock is needed, what input a deep learning model expects, etc.), a bug detection tool can check if the code matches the specification.
Software text, including code comments, API documentation, and user manuals, contains a rich amount of semantic information. Software text can provide a great data source for obtaining programs' correctness information, discovering important problems, and understanding programmers' needs.
What we have done in this direction: We proposed and conducted the first studies to leverage code comments to automatically detect software bugs and bad comments. We achieve these goals by combining techniques from different areas, including natural language processing (NLP), machine learning, information retrieval, program analysis and statistics. We have analyzed various forms of software text using different techniques to address various real-world problems.
(1) cComment: Understanding comments and the potential of utilizing comments. We conduct a comprehensive comment characteristics study on 6 pieces of large software, i.e., Linux, FreeBSD, OpenSolaris, MySQL, Firefox, and Eclipse, which are different types of software (OS, server, and desktop application) and are written in different programming languages (C, C++, and Java). By studying comments written by programmers, we have learned the real needs' of programmers, which can (1) motivate the design of new techniques or improving the usability of the existing tools for improving software reliability, and (2) help developers identify pervasive and important problems and adopt some existing tools or languages for help. We learned many findings including that at least 52.6 ± 2.9% of the comments could be leveraged by existing or to-be-proposed tools for improving reliability.
(2) iComment: Using comments to detect software bugs and bad comments. When comments and code mismatch, it indicates either (1) bugs -- source code does not follow the correct comment, or (2) bad comments -- the comment is wrong or outdated, which can later lead to bugs. iComment takes the first step to detect such comment-code inconsistencies by automatically extracting specifications from comments, and then using flow-sensitive and context-sensitive static program analysis tools to check these specifications against source code. iComment has found 60 previously unknown bugs and bad comments in large software, i.e., Linux, Mozilla, Apache and Wine, and many of them have already been confirmed and fixed by the corresponding developers.
(3) aComment: Mining Annotations from Comments and Code to Detect Interrupt-Related Concurrency Bugs. To detect OS concurrency bugs, we proposed a new type of annotations interrupt related annotations and automatically generated 96,821 such annotations for the Linux kernel with little manual effort. These annotations have been used to automatically detect 9 real OS concurrency bugs (7 are previously unknown). A key technique is using a hybrid approach to extract annotations from both code and comments written in natural language to achieve better coverage and accuracy in annotation extraction and bug detection.
(4) @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies
(5) New program and text analysis for new purposes including guiding symbolic execution to test software, extracting web API specifications, generating code from API documentation, ...