RESTful API Automated Test Case Generation with EvoMaster

ACM Transactions on Software Engineering and Methodology, Journal Year: 2019, Volume and Issue: 28(1), P. 1 - 37

Published: Jan. 9, 2019

RESTful APIs are widespread in industry, especially enterprise applications developed with a microservice architecture. A web service will provide data via an API over the network using HTTP, possibly interacting databases and other services. Testing poses challenges, because inputs/outputs sequences of HTTP requests/responses to remote server. Many approaches literature do black-box testing, often tested is whose code not available. In this article, we consider testing from point view developers, who have full access that they writing. Therefore, propose fully automated white-box approach, where test cases automatically generated evolutionary algorithm. Tests rewarded based on coverage fault-finding metrics. However, REST protocol but rather set guidelines how design resources accessed endpoints. For example, there related should be structured hierarchical URIs different verbs used represent well-defined actions those resources. Test-case generation for only rely information source might able identify create prerequisite needed before being some Smart sampling techniques exploit knowledge best practices generate tests predefined structures speed up search. We implemented our technique tool called E vo M aster , which open source. Experiments five open-source, yet non-trivial, services show novel found 80 real bugs applications. obtained lower than one achieved by manually written suites already existing Research directions further improve such approach therefore discussed, as handling SQL databases.

Language: Английский

---

Testing autonomous cars for feature interaction failures using many-objective search

Published: Aug. 20, 2018

Complex systems such as autonomous cars are typically built a composition of features that independent units functionality. Features tend to interact and impact one another's behavior in unknown ways. A challenge is detect manage feature interactions, particular, those violate system requirements, hence leading failures. In this paper, we propose technique interaction failures by casting problem into search-based test generation problem. We define set hybrid objectives (distance functions) combine traditional coverage-based heuristics with new specifically aimed at revealing develop algorithm, called FITEST, guided our objectives. FITEST extends recently proposed many-objective evolutionary algorithms reduce the time required compute fitness values. evaluate approach using two versions an industrial self-driving system. Our results show able identify more than twice many baseline used software testing literature (i.e., failure-based objectives). Further, feedback from domain experts indicates detected represent real faults their were not previously identified based on analysis requirements.

Language: Английский

---

An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application

Published: May 1, 2017

Automated unit test generation has been extensively studied in the literature recent years. Previous studies on open source systems have shown that tools are quite effective at detecting faults, but how and applicable they an industrial application? In this paper, we investigate question using a life insurance pension products calculator engine owned by SEB Life & Pension Holding AB Riga Branch. To study fault-finding effectiveness, extracted 25 real faults from version history of software project, applied two up-to-date for Java, EVOSUITE RANDOOP, which implement search-based feedback-directed random generation, respectively. Automatically generated suites detected up to 56.40% (EVOSUITE) 38.00% (RANDOOP) these faults. The analysis our results demonstrates challenges need be addressed order improve fault detection tools. particular, classification undetected shows 97.62% them depend either "specific primitive values" (50.00%) or construction "complex state configuration objects" (47.62%). applicability, surveyed developers application under their experience opinions about cases. This leads insights requirements academic prototypes successful technology transfer research practice, such as integrate with popular build tools, readability tests.

Language: Английский

---

The impact of test case summaries on bug fixing performance

Proceedings of the 44th International Conference on Software Engineering, Journal Year: 2016, Volume and Issue: unknown

Published: May 13, 2016

Automated test generation tools have been widely investigated with the goal of reducing cost testing activities. However, generated tests shown not to help developers in detecting and finding more bugs even though they reach higher structural coverage compared manual testing. The main reason is that are difficult understand maintain. Our paper proposes an approach, coined TestDescriber, which automatically generates case summaries portion code exercised by each individual test, thereby improving understandability. We argue this approach can complement current techniques around automated unit or search-based designed generate a possibly minimal set cases. In evaluating our we found (1) find twice as many bugs, (2) significantly improve comprehensibility cases, considered particularly useful developers.

Language: Английский

---

Automated unit test generation for classes with environment dependencies

Published: Sept. 15, 2014

Automated test generation for object-oriented software typically consists of producing sequences calls aiming at high code coverage. In practice, the success this process may be inhibited when classes interact with their environment, such as file system, network, user-interactions, etc. This leads to two major problems: First, that depends on environment can sometimes not fully covered simply by generating a class under test, example execution branch contents file. Second, even if is environment-dependent covered, resulting tests unstable, i.e., they would pass first generated, but then fail executed in different environment. For example, make use system time have failing assertions are than were generated.

Language: Английский

---

Continuous test generation

Published: Sept. 15, 2014

In object oriented software development, automated unit test generation tools typically target one class at a time. A class, however, is usually part of project consisting more than and these are subject to changes over This context offers significant potential improve for individual classes. this paper, we introduce Continuous Test Generation (CTG), which includes during continuous integration (i.e., infrastructure that regularly builds tests projects). CTG several benefits: First, it answers the question how much time spend on each in project. Second, helps decide order them. Finally, classes should be subjected first place. We have implemented using EvoSuite tool, performed experiments eight most popular open source projects available GitHub, ten randomly selected from SF100 corpus, five industrial projects. Our demonstrate improvements up +58% branch coverage +69% thrown undeclared exceptions, while reducing spent by +83%.

Language: Английский

---

Automatic test case generation: what if test code quality matters?

Published: July 7, 2016

Test case generation tools that optimize code coverage have been extensively investigated. Recently, researchers suggested to add other non-coverage criteria, such as memory consumption or readability, increase the practical usefulness of generated tests. In this paper, we observe test quality metrics, and cohesion coupling in particular, are valuable candidates additional criteria. Indeed, tests with low and/or high shown a negative impact on future maintenance activities. an exploratory investigation show most indeed affected by poor quality. For reason, incorporate metrics into main loop search-based algorithm for generation. Through empirical study our approach is not only able generate more cohesive less coupled, but can (i) branch up 10% when enough time given search (ii) result statistically shorter

Language: Английский

---

Combining symbolic execution and search-based testing for programs with complex heap inputs

Published: July 10, 2017

Despite the recent improvements in automatic test case generation, handling complex data structures as inputs is still an open problem. Search-based approaches can generate sequences of method calls that instantiate structured to exercise a relevant portion code, but fall short building execute program elements whose reachability determined by structural features input themselves. Symbolic execution techniques effectively handle inputs, do not identify through legal interfaces. In this paper, we propose new approach automatically cases for programs with inputs. We use symbolic path conditions characterise dependencies between paths and structures, convert optimisation problems solve search-based produce those Our preliminary results show indeed effective generating thus opening promising research direction.

Language: Английский

---

RESTful API Automated Test Case Generation

Published: July 1, 2017

Nowadays, web services play a major role in the development of enterprise applications. Many such applications are now developed using service-oriented architecture (SOA), where microservices is one its most popular kind. A RESTful service will provide data via an API over network HTTP, possibly interacting with databases and other services. Testing poses challenges, as inputs/outputs sequences HTTP requests/responses to remote server. approaches literature do black-box testing, tested whose code not available. In this paper, we consider testing from point view developers, which have full access that they writing. Therefore, propose fully automated white-box approach, test cases automatically generated evolutionary algorithm. Tests rewarded based on coverage fault finding metrics. We implemented our technique tool called EVOMASTER, open-source. Experiments two open-source, yet non-trivial industrial one, show novel did find 38 real bugs those However, obtained lower than achieved by manually written suites already existing Research directions how further improve approach therefore discussed.

Language: Английский

---

A Guided Genetic Algorithm for Automated Crash Reproduction

Published: May 1, 2017

To reduce the effort developers have to make for crash debugging, researchers proposed several solutions automatic failure reproduction. Recent advances use of symbolic execution, mutation analysis, and directed model checking as underling techniques post-failure analysis stack traces. However, existing approaches still cannot reproduce many real-world crashes due such limitations environment dependencies, path explosion, time complexity. address these challenges, we present EvoCrash, a approach which uses novel Guided Genetic Algorithm (GGA) cope with large search space characterizing software programs. Our empirical study on three open-source systems shows that EvoCrash can replicate 41 (82%) crashes, 34 (89%) are useful reproductions debugging purposes, outperforming state-of-the-art in replication.

Language: Английский

---