Our customer, a leading Operating System vendor, needed to generate compatibility impact data for third party device drivers for devices across various device categories with an upcoming version of their operating system with a customer-centric usage view.
with several thousand devices to support it was important to discover a strategic direction for device compatibility testing. This had to be accomplished in a short time and within reasonable budgets. Obviously the challenge was to select a right sized sample broad enough to be representative and small enough to be affordable.
The required workflow applications such as benefits enrollment etc. could not be specified completely up-front, but would get added to the system on an on-going basis. What was needed was a framework for adding such applications.
Multiple devices across several device categories needed to be set up, their feature and functionality understood, tests designed, executed and the test result data recorded.
In a kick off meeting hosted by Harbinger Systems, both parties detailed out the requirements and goals in greater depth. Harbinger then went about designing and setting up a test laboratory. The lab setup consisted of test machines across different hardware platforms and debuggers. Harbinger then put a test design and execution strategy in place. The strategy consisted of prioritizing testing into different scenarios like device and driver management, basic functionality, extended functionality, power management and testing of applets / utilities. Harbinger also played a role in device selection which was based on a wide number of criteria.
Through the coverage of a wide range of popular devices for basic user scenarios, significant actionable data was uncovered. This data provided insights into any breaking changes in the new OS version, resulting in a better understanding of the direction that further device compatibility testing should take.