QTEP: Quality-Aware Test Case Prioritization

Test case prioritization (TCP) is a practical activity in software testing for exposing faults earlier. Researchers have proposed many TCP techniques to re-order test cases. Among them, coverage-based TCPs have been widely investigated. Specifically, coverage-based TCP approaches leverage coverage information between source code and test cases, i.e., static code coverage and dynamic code coverage, to schedule test cases. Existing coverage-based TCP techniques mainly focus on maximizing coverage while often do not consider the likely distribution of faults in source code. However, software faults are not often equally distributed in source code, e.g., around 80% faults are located in about 20% source code. Intuitively, test cases that cover the faulty source code should have higher priorities, since they are more likely to find faults.

QTEP is proposed to address the limitation of existing coverage-based TCP algorithms. In QTEP, we leverage code inspection techniques, i.e., a typical statistic defect prediction model and a typical static bug finder, to detect fault-prone source code and then adapt existing coverage-based TCP algorithms by considering the weighted source code in terms of fault-proneness.

Paper:

Song Wang, Jaechang Nam, and Lin Tan. "QTEP: Quality-aware Test Case Prioritization," 11th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. (FSE 2017) [download]

Experimental Data and Additional Results:

Projects and faults:

a) Coverage information [download]

b) Real faults for regression cases [download]

c) Mutation faults for new test cases [download]

d) Metrics for CLAIM [download]

e) FindBugs reports [download]

Additional Result (Comparison between dynamic coverage based QTEP and the corresponding traditional TCPs):

Porject	Scenario	DMT	DMT-DP	DMT-BF	DMA	DMA-DP	DMA-BF	DST	DST-DP	DST-BF	DSA	DSA-DP	DSA-BF
time&money	C-R	0.5	0.65	0.75	0.27	0.34	0.43	0.84	0.84	0.9	0.9	0.9	0.94
	M-R	0.47	0.62	0.73	0.59	0.59	0.59	0.55	0.59	0.57	0.59	0.59	0.59
	C-RN	0.25	0.31	0.3	0.19	0.21	0.22	0.42	0.42	0.44	0.43	0.43	0.44
	M-RN	0.18	0.24	0.27	0.29	0.29	0.29	0.27	0.29	0.27	0.28	0.28	0.28
mime4J	C-R	0.71	0.73	0.71	0.64	0.82	0.74	0.89	0.9	0.92	0.86	0.86	0.88
	M-R	0.68	0.77	0.78	0.63	0.63	0.63	0.72	0.79	0.84	0.55	0.55	0.55
	C-RN	0.25	0.31	0.3	0.19	0.21	0.22	0.42	0.42	0.44	0.43	0.43	0.44
	M-RN	0.18	0.24	0.27	0.29	0.29	0.29	0.27	0.29	0.27	0.28	0.28	0.28
jaxen	C-R	0.6	0.63	0.63	0.49	0.74	0.55	0.79	0.8	0.8	0.76	0.79	0.8
	M-R	0.47	0.51	0.55	0.82	0.84	0.84	0.63	0.64	0.66	0.52	0.53	0.54
	C-RN	0.34	0.34	0.35	0.35	0.42	0.35	0.44	0.43	0.46	0.42	0.42	0.45
	M-RN	0.14	0.14	0.14	0.33	0.33	0.33	0.37	0.37	0.37	0.35	0.35	0.36
xml-security	C-R	0.37	0.37	0.37	0.5	0.5	0.5	0.37	0.5	0.37	0.63	0.7	0.63
	M-R	0.33	0.36	0.46	0.26	0.28	0.26	0.34	0.45	0.36	0.26	0.26	0.26
	C-RN	0.2	0.2	0.2	0.27	0.27	0.27	0.2	0.27	0.23	0.33	0.33	0.33
	M-RN	0.27	0.27	0.35	0.21	0.22	0.21	0.28	0.36	0.29	0.21	0.21	0.21
xstream	C-R	0.62	0.63	0.62	0.44	0.48	0.48	0.8	0.81	0.82	0.73	0.78	0.76
	M-R	0.69	0.71	0.71	0.59	0.69	0.59	0.72	0.77	0.77	0.45	0.45	0.45
	C-RN	0.57	0.57	0.57	0.47	0.48	0.48	0.64	0.64	0.64	0.65	0.66	0.66
	M-RN	0.3	0.39	0.3	0.28	0.28	0.28	0.32	0.33	0.34	0.22	0.24	0.25
commons-lang	C-R	0.9	0.91	0.91	0.87	0.9	0.89	0.79	0.79	0.82	0.86	0.86	0.89
	M-R	0.91	0.92	0.93	0.36	0.37	0.36	0.48	0.48	0.53	0.72	0.72	0.72
	C-RN	0.39	0.39	0.4	0.42	0.42	0.42	0.56	0.56	0.57	0.56	0.56	0.56
	M-RN	0.42	0.43	0.43	0.21	0.21	0.21	0.21	0.21	0.24	0.31	0.35	0.37
joda-time	C-R	0.53	0.56	0.59	0.52	0.61	0.68	0.6	0.63	0.62	0.63	0.66	0.66
	M-R	0.48	0.51	0.53	0.68	0.74	0.74	0.42	0.44	0.47	0.69	0.69	0.69
	C-RN	0.55	0.57	0.59	0.57	0.59	0.63	0.65	0.6	0.68	0.63	0.63	0.63
	M-RN	0.18	0.2	0.2	0.24	0.24	0.24	0.18	0.19	0.19	0.27	0.28	0.28

Team:

Song Wang (song.wang@uwaterloo.ca)

JC Nam (jc.nam@uwaterloo.ca)

Lin Tan (lintan@purdue.edu)

Send us an email if you have any questions about this work.

License and Disclaimer