Data Driven testing for OS API
Harbinger's customer, a leading Operating System vendor, wanted to perform robust tests on one of its core API libraries "already considered to be stable". With a growing number of OS services leveraging the core API, the client needed an enhanced measure of confidence in its stability from a security, performance and functionality perspective.
This project was a cut above regular OS testing projects. It not only required an in-depth understanding of the white box techniques for API testing but also demanded innovations in reducing the number of tests without compromising coverage.
Harbinger used its extensive knowledge and innovative approach of API testing in this project. Harbinger proposed a two-phase approach for each API function within the library followed by a data-driven test execution approach.
The test design approach consisted of a black-box analysis followed by a white-box analysis for each API function. The black box analysis phase consisted of subjecting the API to parameterized analysis through an equivalence class, boundary-value analysis approach to derive valid and invalid data for API function parameters. This generated an explosion of parameter combinations, to which Harbinger applied innovative elimination strategies to reduce the combinations needed to be tested. The white-box analysis phase consisted of techniques like decision / condition coverage techniques to ensure that all possible execution paths within the source code are executed. An effective technique to combine black-box analysis with strong white-box analysis capabilities resulted in excellent tests from a security, performance and functionality standpoint.
The test execution approach taken by Harbinger involved the design and development of a data-driven test harness. This data-driven test harness was designed to abstract the test data from the API function calls and further abstract the API function calls from the result validation. This abstraction allowed for the ability to easily add / modify / delete test cases.
The customer received a set of test cases that thoroughly tested the API library for valid, invalid, special, performance and security scenarios with room for easy addition / modification and deletion.