Lecture notes in computer science, Journal Year: 2023, Volume and Issue: unknown, P. 108 - 122
Published: Dec. 3, 2023
Language: Английский
Published: July 15, 2022
Modern web services routinely provide REST APIs for clients to access their functionality. These present unique challenges and opportunities automated testing, driving the recent development of many techniques tools that generate test cases API endpoints using various strategies. Understanding how these compare one another is difficult, as they have been evaluated on different benchmarks metrics. To fill this gap, we performed an empirical study aimed understand landscape in testing guide future research area. We first identified, through a systematic selection process, set 10 state-of-the-art included developed by both researchers practitioners. then applied benchmark 20 real-world open-source RESTful analyzed performance terms code coverage achieved failures triggered. This analysis allowed us identify strengths, weaknesses, limitations considered underlying strategies, well implications our findings
Language: Английский
Citations
39
ACM Transactions on Software Engineering and Methodology, Journal Year: 2023, Volume and Issue: 33(1), P. 1 - 41
Published: Aug. 21, 2023
In industry, RESTful APIs are widely used to build modern Cloud Applications. Testing them is challenging, because not only do they rely on network communications, but also deal with external services like databases. Therefore, there has been a large amount of research sprout in recent years how automatically verify this kind web services. article, we present comprehensive review the current state-of-the-art testing based analysis 92 scientific articles. These articles were gathered by utilizing search queries formulated around concept API seven popular We eliminated irrelevant our predefined criteria and conducted snowballing phase minimize possibility missing any relevant paper. This survey categorizes summarizes existing work discusses challenges verification APIs. clearly shows an increasing interest among researchers field, from 2017 onward. However, still lot open overcome.
Language: Английский
Citations
30
Proceedings of the 44th International Conference on Software Engineering, Journal Year: 2022, Volume and Issue: unknown
Published: May 21, 2022
This paper presents RestCT, a systematic and fully automatic approach that adopts Combinatorial Testing (CT) to test RESTful APIs. RestCT is in it covers tests not only the interactions of certain number operations APIs, but also particular input-parameters every single operation. realised by novel two-phase case generation approach, which first generates constrained sequence covering array determine execution orders available operations, then applies an adaptive strategy generate refine several arrays concretise each its application relies on given Swagger specification The creation CT models (especially, inferring dependency relationships both input-parameters), cases are performed without any human intervention. Experimental results 11 real-world APIs demonstrate effectiveness efficiency RestCT. In particular, can find eight new bugs, where one them be triggered state-of-the-art testing tool
Language: Английский
Citations
29
ACM Transactions on Software Engineering and Methodology, Journal Year: 2023, Volume and Issue: 32(6), P. 1 - 45
Published: May 13, 2023
RESTful APIs are a type of web service that widely used in industry. In the past few years, lot effort research community has been spent designing novel techniques to automatically fuzz those find faults them. Many real were found large variety APIs. However, usually analyzed fuzzers treat as black-box, and no analysis what is actually covered these systems done. Therefore, although clearly useful for practitioners, we do not know their current limitations actual effectiveness. Solving this necessary step be able design better, more efficient, effective techniques. To address issue, article compare seven state-of-the-art on 18 open source—1 industrial 1 artificial—RESTful We then analyze source code which parts fail generate tests. This points clear fuzzers, listing concrete follow-up challenges community.
Language: Английский
Citations
21
ACM Transactions on Software Engineering and Methodology, Journal Year: 2023, Volume and Issue: 32(5), P. 1 - 38
Published: Feb. 23, 2023
Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over network. RPC widely applied in industry for building large-scale distributed systems, such as Microservices. Modern frameworks include, example, Thrift, gRPC, SOFARPC, and Dubbo. Testing systems using communications very challenging, due the complexity of various system could employ. To best our knowledge, there does not exist any tool or solution that enable automated testing modern RPC-based services. fill this gap, article we propose first approach literature, together with an open source tool, fuzzing APIs. The context white-box search-based techniques. tackle schema extraction frameworks, formulate specification along parser allows from code JVM Then, extracted employ search produce tests by maximizing heuristics newly defined specific domain. We built extension fuzzer (i.e., EvoMaster ), has been integrated into real industrial pipeline be development process assess novel approach, conducted empirical study two artificial four web selected partner. In addition, further demonstrate its effectiveness application settings, report results employing another 50 APIs autonomously partner their processes. Results show capable enabling test case generation 2 54 industrial). also compared simple gray-box technique existing manually written tests. Our achieves significant improvements on coverage. Regarding fault detection, conducting careful review generated APIs, total 41 faults were identified, which have now fixed. Another 8,377 detected are currently under investigation.
Language: Английский
Citations
20
Published: Feb. 20, 2025
Language: Английский
Citations
0
ACM Transactions on Software Engineering and Methodology, Journal Year: 2025, Volume and Issue: unknown
Published: April 23, 2025
Testing large and complex enterprise software systems can be a challenging task. This is especially the case when functionality of system depends on interactions with other external services over network (e.g., web accessed through REST API calls). Although several techniques in research literature have been shown to effective at generating test cases good number different testing contexts, dealing still major challenge. In industry, common approach mock for purposes. However, configuring very time-consuming task, e.g., may not under control same developers tested application, making it identify simulate various possible responses. this paper, we present novel search-based aimed fully automated mocking as part white-box, fuzzing. We rely code instrumentation detect all services, how their response data parsed. then use such information enhance The application automatically modified (by manipulating DNS lookups) rather interact instances servers. search process only generates inputs applications but also configures responses those server instances, aiming maximizing coverage fault-finding. An empirical study four open-source APIs from EMB, one industrial an industry partner, shows effectiveness our (i.e., terms line fault detection).
Language: Английский
Citations
0
ACM Transactions on Software Engineering and Methodology, Journal Year: 2024, Volume and Issue: 33(6), P. 1 - 36
Published: March 11, 2024
Due to its importance and widespread use in industry, automated testing of REST APIs has attracted major interest from the research community last few years. However, most work literature been focused on black-box fuzzing. Although existing fuzzers have used automatically find many faults APIs, there are still several open challenges that hinder achievement better results (e.g., terms code coverage fault finding). For example, under-specified schemas a issue for fuzzers. Currently, EvoMaster is only tool supports white-box fuzzing APIs. In this paper, we provide series novel heuristics, including example how deal with constrains API schemas, as well SQL databases. Our techniques implemented an extension our open-source, search-based fuzzer . An empirical study 14 EMB corpus, plus one industrial API, shows clear improvements some these
Language: Английский
Citations
3
Science of Computer Programming, Journal Year: 2025, Volume and Issue: unknown, P. 103322 - 103322
Published: May 1, 2025
Language: Английский
Citations
0
Software Quality Journal, Journal Year: 2023, Volume and Issue: 31(3), P. 947 - 990
Published: March 6, 2023
Abstract Research in software testing often involves the development of prototypes. Like any piece software, there are challenges development, use and verification such tools. However, some rather specific to this problem domain. For example, these tools developed by PhD students straight out bachelor/master degrees, possibly lacking industrial experience development. Prototype used carry empirical studies, studying different parameters novel designed algorithms. Software scaffolding is needed run large sets experiments efficiently. Furthermore, when using AI-based techniques like evolutionary algorithms, care needs be taken deal with their randomness, which further complicates verification. The aforementioned represent we have identified for In paper, report on our building open-source EvoMaster tool, aims at system-level test case generation enterprise applications. Many faced would common researcher needing build tool Therefore, one goal that shared here will boost research community, providing concrete solutions many kind Ultimately, lead increase impact scientific practice.
Language: Английский
Citations
8