Testing RESTful APIs: A Survey

ACM Transactions on Software Engineering and Methodology, Journal Year: 2023, Volume and Issue: 33(1), P. 1 - 41

Published: Aug. 21, 2023

In industry, RESTful APIs are widely used to build modern Cloud Applications. Testing them is challenging, because not only do they rely on network communications, but also deal with external services like databases. Therefore, there has been a large amount of research sprout in recent years how automatically verify this kind web services. article, we present comprehensive review the current state-of-the-art testing based analysis 92 scientific articles. These articles were gathered by utilizing search queries formulated around concept API seven popular We eliminated irrelevant our predefined criteria and conducted snowballing phase minimize possibility missing any relevant paper. This survey categorizes summarizes existing work discusses challenges verification APIs. clearly shows an increasing interest among researchers field, from 2017 onward. However, still lot open overcome.

Language: Английский

---

Combinatorial testing of RESTful APIs

Proceedings of the 44th International Conference on Software Engineering, Journal Year: 2022, Volume and Issue: unknown

Published: May 21, 2022

This paper presents RestCT, a systematic and fully automatic approach that adopts Combinatorial Testing (CT) to test RESTful APIs. RestCT is in it covers tests not only the interactions of certain number operations APIs, but also particular input-parameters every single operation. realised by novel two-phase case generation approach, which first generates constrained sequence covering array determine execution orders available operations, then applies an adaptive strategy generate refine several arrays concretise each its application relies on given Swagger specification The creation CT models (especially, inferring dependency relationships both input-parameters), cases are performed without any human intervention. Experimental results 11 real-world APIs demonstrate effectiveness efficiency RestCT. In particular, can find eight new bugs, where one them be triggered state-of-the-art testing tool

Language: Английский

---

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

ACM Transactions on Software Engineering and Methodology, Journal Year: 2023, Volume and Issue: 32(6), P. 1 - 45

Published: May 13, 2023

RESTful APIs are a type of web service that widely used in industry. In the past few years, lot effort research community has been spent designing novel techniques to automatically fuzz those find faults them. Many real were found large variety APIs. However, usually analyzed fuzzers treat as black-box, and no analysis what is actually covered these systems done. Therefore, although clearly useful for practitioners, we do not know their current limitations actual effectiveness. Solving this necessary step be able design better, more efficient, effective techniques. To address issue, article compare seven state-of-the-art on 18 open source—1 industrial 1 artificial—RESTful We then analyze source code which parts fail generate tests. This points clear fuzzers, listing concrete follow-up challenges community.

Language: Английский

---

White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study

ACM Transactions on Software Engineering and Methodology, Journal Year: 2023, Volume and Issue: 32(5), P. 1 - 38

Published: Feb. 23, 2023

Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over network. RPC widely applied in industry for building large-scale distributed systems, such as Microservices. Modern frameworks include, example, Thrift, gRPC, SOFARPC, and Dubbo. Testing systems using communications very challenging, due the complexity of various system could employ. To best our knowledge, there does not exist any tool or solution that enable automated testing modern RPC-based services. fill this gap, article we propose first approach literature, together with an open source tool, fuzzing APIs. The context white-box search-based techniques. tackle schema extraction frameworks, formulate specification along parser allows from code JVM Then, extracted employ search produce tests by maximizing heuristics newly defined specific domain. We built extension fuzzer (i.e., EvoMaster ), has been integrated into real industrial pipeline be development process assess novel approach, conducted empirical study two artificial four web selected partner. In addition, further demonstrate its effectiveness application settings, report results employing another 50 APIs autonomously partner their processes. Results show capable enabling test case generation 2 54 industrial). also compared simple gray-box technique existing manually written tests. Our achieves significant improvements on coverage. Regarding fault detection, conducting careful review generated APIs, total 41 faults were identified, which have now fixed. Another 8,377 detected are currently under investigation.

Language: Английский

---

Handling Web Service Interactions in Fuzzing with Search-Based Mock-Generation

ACM Transactions on Software Engineering and Methodology, Journal Year: 2025, Volume and Issue: unknown

Published: April 23, 2025

Testing large and complex enterprise software systems can be a challenging task. This is especially the case when functionality of system depends on interactions with other external services over network (e.g., web accessed through REST API calls). Although several techniques in research literature have been shown to effective at generating test cases good number different testing contexts, dealing still major challenge. In industry, common approach mock for purposes. However, configuring very time-consuming task, e.g., may not under control same developers tested application, making it identify simulate various possible responses. this paper, we present novel search-based aimed fully automated mocking as part white-box, fuzzing. We rely code instrumentation detect all services, how their response data parsed. then use such information enhance The application automatically modified (by manipulating DNS lookups) rather interact instances servers. search process only generates inputs applications but also configures responses those server instances, aiming maximizing coverage fault-finding. An empirical study four open-source APIs from EMB, one industrial an industry partner, shows effectiveness our (i.e., terms line fault detection).

Language: Английский

---

Resource and dependency based test case generation for RESTful Web services

Empirical Software Engineering, Journal Year: 2021, Volume and Issue: 26(4)

Published: June 2, 2021

Abstract Nowadays, RESTful web services are widely used for building enterprise applications. REST is not a protocol, but rather it defines set of guidelines on how to design APIs access and manipulate resources using HTTP over network. In this paper, we propose an enhanced search-based method automated system test generation services, by exploiting domain knowledge the handling resources. The proposed techniques use specific effective templates structure actions (i.e., ordered sequences calls) within individual in evolutionary search. action developed based semantics methods services’ addition, five novel sampling strategies with four resource-based sampling) cases that can one or more these templates. further supported new, specialized mutation operators mutation) search take into account generated cases. Moreover, dependency detect possible dependencies among tested mutations then information detected dependencies. To evaluate our approach, implemented as extension EvoMaster tool, conducted empirical study two selected baselines 7 open-source 12 synthetic services. Results show approach obtains significant improvement performance baselines, e.g., up + 130.7% relative (growing from 27.9% 64.3%) line coverage.

Language: Английский

---

Enhancing Search-based Testing with Testability Transformations for Existing APIs

ACM Transactions on Software Engineering and Methodology, Journal Year: 2021, Volume and Issue: 31(1), P. 1 - 34

Published: Sept. 28, 2021

Search-based software testing (SBST) has been shown to be an effective technique generate test cases automatically. Its effectiveness strongly depends on the guidance of fitness function. Unfortunately, a common issue in SBST is so-called flag problem , where landscape presents plateau that provides no search. In this article, we provide series novel testability transformations aimed at providing context commonly used API calls (e.g., strings need converted into valid date/time objects). We also specific helping REST Web Services. implemented our techniques as extension EvoMaster tool generates system-level cases. Experiments nine open-source web services, well industrial service, show improve performance significantly.

Language: Английский

---

Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster

ACM Transactions on Software Engineering and Methodology, Journal Year: 2021, Volume and Issue: 31(1), P. 1 - 52

Published: Sept. 28, 2021

REST web services are widely popular in industry, and search techniques have been successfully used to automatically generate system-level test cases for those systems. In this article, we propose a novel mutation operator which is designed specifically generation at system-level, with particular focus on APIs. API testing, often system testing general, an individual can long complex chromosome. Furthermore, there two specific issues: (1) fitness evaluation highly costly compared the number of objectives (e.g., targets) optimize for; (2) large part genotype might no impact phenotype individuals input data that has execution flow tested program). Due these issues, it be not suitable apply typical low rate like 1/ n (where genes individual), would lead mutating only one gene average. Therefore, adaptive weight-based hypermutation, aware different characteristics mutated genes. We developed strategies enable selection adaptively based their history throughout search. To assess our proposed operator, implemented EvoMaster tool, integrated MIO algorithm, further conducted empirical study three artificial APIs four real-world Results show demonstrates noticeable improvements over default MIO. It provides significant improvement performance six out seven case studies, where relative up +12.09% target coverage, +12.69% line +32.51% branch coverage.

Language: Английский

---

On the Faults Found in REST APIs by Automated Test Generation

ACM Transactions on Software Engineering and Methodology, Journal Year: 2022, Volume and Issue: 31(3), P. 1 - 43

Published: March 7, 2022

RESTful web services are often used for building a wide variety of enterprise applications. The diversity and increased number applications using APIs means that increasing amounts resources spent developing testing these systems. Automation in test data generation provides useful way generating fast efficient manner. However, automated results large suites hard to evaluate investigate manually. This article proposes taxonomy the faults we have found search-based software techniques applied on APIs. is first step understanding, analyzing, ultimately fixing We propose apply density-based clustering algorithm cases evolved during search allow better separation between different groups faults. needed enable engineers highlight focus most serious Tests were automatically generated set eight case studies, seven open-source one industrial. clustered based reported last executed line error messages returned, when such available. tests manually evaluated determine their root causes obtain additional information. presents manual analysis 415 studies method support classification resulting cases.

Language: Английский

---

EMB: A Curated Corpus of Web/Enterprise Applications And Library Support for Software Testing Research

Published: April 1, 2023

Web Services like REST, GraphQL and RPC APIs are widely used in industry. They form the backends of modern Cloud Applications. In recent years, there has been an increase interest research community about fuzzing web services. However, is no clear, common benchmark literature that can be for comparing techniques ease experimentation. Even if nowadays it not so difficult to find services on open-source repositories such as GitHub, quite a bit work might required setup databases authentication information (e.g., hashed passwords). Furthermore, how start stop applications vary greatly among different frameworks Spring DropWizard) implement For all these reasons, since 2017 we have created maintained corpus called EMB, together with tooling configurations needed run software testing experiments. Originally, EMB was evaluating fuzzer EvoMaster, but (and been) by other tools/researchers well. This paper discusses designed its libraries experiments APIs. An introductory video currently accessed at https://youtu.be/wJs34ATgLEw

Language: Английский

---