Whole Test Suite Generation

IEEE Transactions on Software Engineering, Journal Year: 2012, Volume and Issue: 39(2), P. 276 - 291

Published: Feb. 23, 2012

Not all bugs lead to program crashes, and not always is there a formal specification check the correctness of software test's outcome. A common scenario in testing therefore that test data are generated, tester manually adds oracles. As this difficult task, it important produce small yet representative sets, representativeness typically measured using code coverage. There is, however, fundamental problem with approach targeting one coverage goal at time: Coverage goals independent, equally difficult, sometimes infeasible—the result generation dependent on order how many them feasible. To overcome problem, we propose novel paradigm which whole suites evolved aim covering same time while keeping total size as possible. This has several advantages, for example, its effectiveness affected by number infeasible targets code. We have implemented EvoSuite tool, compared addressing time. Evaluated open source libraries an industrial case study 1,741 classes, show achieved up 188 times branch traditional single branches, 62 percent smaller suites.

Language: Английский

---

A study of equivalent and stubborn mutation operators using human analysis of equivalence

Proceedings of the 44th International Conference on Software Engineering, Journal Year: 2014, Volume and Issue: unknown, P. 919 - 930

Published: May 20, 2014

Though mutation testing has been widely studied for more than thirty years, the prevalence and properties of equivalent mutants remain largely unknown. We report on causes their relationship to stubborn (those that undetected by a high quality test suite, yet are non-equivalent). Our results, based manual analysis 1,230 from 18 programs, reveal highly uneven distribution equivalence stubbornness. For example, ABS class half UOI generate many almost no mutants, while LCR generates few mutants. conclude previous effectiveness studies fault seeding could be skewed, developers tools should prioritise those operators we found disproportionately (and equivalent)

Language: Английский

---

Testing autonomous cars for feature interaction failures using many-objective search

Published: Aug. 20, 2018

Complex systems such as autonomous cars are typically built a composition of features that independent units functionality. Features tend to interact and impact one another's behavior in unknown ways. A challenge is detect manage feature interactions, particular, those violate system requirements, hence leading failures. In this paper, we propose technique interaction failures by casting problem into search-based test generation problem. We define set hybrid objectives (distance functions) combine traditional coverage-based heuristics with new specifically aimed at revealing develop algorithm, called FITEST, guided our objectives. FITEST extends recently proposed many-objective evolutionary algorithms reduce the time required compute fitness values. evaluate approach using two versions an industrial self-driving system. Our results show able identify more than twice many baseline used software testing literature (i.e., failure-based objectives). Further, feedback from domain experts indicates detected represent real faults their were not previously identified based on analysis requirements.

Language: Английский

---

Evolutionary Generation of Whole Test Suites

Published: July 1, 2011

Recent advances in software testing allow automatic derivation of tests that reach almost any desired point the source code. There is, however, a fundamental problem with general idea targeting one distinct test coverage goal at time: Coverage goals are neither independent each other, nor is generation for particular guaranteed to succeed. We present EVOSUITE, search-based approach optimizes whole suites towards satisfying criterion, rather than generating cases directed goals. Evaluated on five open libraries and an industrial case study, we show EVOSUITE achieves up 18 times traditional single branches, 44% smaller suites.

Language: Английский

---

Generating Test Data from OCL Constraints with Search Techniques

IEEE Transactions on Software Engineering, Journal Year: 2013, Volume and Issue: 39(10), P. 1376 - 1402

Published: April 1, 2013

Model-based testing (MBT) aims at automated, scalable, and systematic solutions for complex industrial software systems. To increase chances of adoption in contexts, systems can be modeled using well-established standards such as the Unified Modeling Language (UML) Object Constraint (OCL). Given that test data generation is one major challenges to automate MBT, we focus on from OCL constraints this paper. This endeavor all more challenging given numerous constructs operations are designed facilitate definition constraints. Though search-based has been applied white-box (e.g., branch coverage), its application MBT limited. In paper, propose a set search heuristics targeted guide applications. We evaluate these three algorithms: Genetic Algorithm, (1+1) Evolutionary Alternating Variable Method. empirically our artificial problems, followed by empirical analyses feasibility approach system context robustness testing. Our also compared with most widely referenced solver (UMLtoCSP) literature shows significantly efficient.

Language: Английский

---

Achieving scalable mutation-based generation of whole test suites

Empirical Software Engineering, Journal Year: 2014, Volume and Issue: 20(3), P. 783 - 812

Published: Feb. 8, 2014

Language: Английский

---

A detailed investigation of the effectiveness of whole test suite generation

Empirical Software Engineering, Journal Year: 2016, Volume and Issue: 22(2), P. 852 - 893

Published: March 19, 2016

A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one case at time each these individually, whole suite generation optimizes entire suites towards satisfying the same time. There evidence that overall achieved with this approach superior targeting individual goals. Nevertheless, there remains some uncertainty on (a) whether results generalize beyond branch coverage, (b) might be inferior more focused search particular goals, and (c) could optimized only not already covered. In paper, we perform an in-depth analysis study questions. An empirical 100 Java classes using three different criteria reveals indeed are covered traditional approach, although their number very small in comparison those which exclusively approach. We find keeping archive along tests covering them focusing uncovered overcomes drawback larger classes, leading improved effectiveness generation.

Language: Английский

---

An empirical study of automated unit test generation for Python

Empirical Software Engineering, Journal Year: 2023, Volume and Issue: 28(2)

Published: Jan. 31, 2023

Abstract Various mature automated test generation tools exist for statically typed programming languages such as Java. Automatically generating unit tests dynamically Python, however, is substantially more difficult due to the dynamic nature of these well lack type information. Our Pynguin framework provides Python. In this paper, we extend our previous work on support aspects Python language, and by studying a larger variety well-established state art test-generation algorithms, namely DynaMOSA, MIO, MOSA. Furthermore, improved tool generate regression assertions, whose quality also evaluate. experiments confirm that evolutionary algorithms can outperform random in context similar Java world, DynaMOSA yields highest coverage results. However, results demonstrate there are still fundamental remaining issues, inferring information code without information, currently limiting effectiveness

Language: Английский

---

1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite

Empirical Software Engineering, Journal Year: 2013, Volume and Issue: 20(3), P. 611 - 639

Published: Nov. 14, 2013

Language: Английский

---

Combining Multiple Coverage Criteria in Search-Based Unit Test Generation

Lecture notes in computer science, Journal Year: 2015, Volume and Issue: unknown, P. 93 - 108

Published: Jan. 1, 2015

Language: Английский

---