Revisiting the Relationship Between Fault Detection,Test Adequacy Criteria, and Test Set Size

(2020) Revisiting the Relationship Between Fault Detection,Test Adequacy Criteria, and Test Set Size.

Preview	Text chen2020revisiting.pdf Download (2MB) | Preview
	Other (Plain Text Bibliography) bibliography.txt - Bibliography Download (7kB)

Abstract

The research community has long recognized a complex interrelationship between test set size, test adequacy criteria, and test effectiveness in terms of fault detection. However, there is substantial confusion about the role and importance of controlling for test set size when assessing and comparing test adequacy criteria. This paper makes the following contributions: (1) A review of contradictory analyses of the relationship between fault detection, test suite size, and test adequacy criteria. Specifically, this paper addresses the supposed contradiction of prior work and explains why test suite size is neither a confounding variable, as previously suggested,nor an independent variable that should be experimentally manipulated. (2) An explication and discussion of the experimental design and sampling strategies of prior work, together with a discussion of conceptual and statistical problems, and specific guidelines for future work. (3) A methodology for comparing test-adequacy criteria on an equal basis, which accounts for test suite size by treating it as a covariate. (4) An empirical evaluation that compares the effectiveness of coverage-based and mutation-based testing to one another and random testing. Additionally, this paper proposes probabilistic coupling, a methodology for approximating the representativeness of a set of test goals for a given set of real faults

Item Type:	Conference or Workshop Item (A Paper) (Paper)
Divisions:	Andreas Zeller (Software Engineering, ST)
Conference:	ASE Automated Software Engineering Conference
Depositing User:	Rahul Gopinath
Date Deposited:	01 Dec 2020 10:20
Last Modified:	01 Dec 2020 10:24
Primary Research Area:	NRA4: Secure Mobile and Autonomous Systems
URI:	https://publications.cispa.saarland/id/eprint/3263

Actions

Actions (login required)

View Item