Performance Testing a SaaS Enterprise Product

Situation

Harbinger's customer is a medium-sized business that provides employee benefits administration product to other medium sized businesses. The software is delivered as a service (SaaS) from the company infrastructure and is used by employees of its customer companies. Seeing the growth potential of the market and the availability of modern online marketing channels, the company forecasted a supernormal growth in its total user base and average employee strength of its customers.

Challenge

Despite the glowing forecast, the company's existing product had limited capacity to manage concurrent users in the system. The benefits insurance products industry is used to very high annual spikes in product usage in what is called the Employee Enrollment Period. Addition of few big-ticket customers would only exacerbate the problem of high load at peak time. The company approached Harbinger Systems to analyze the product for its readiness to take on more concurrent and total users.

Harbinger Solution

Harbinger first focused on pages that users accessed most, and carried out load and capacity testing. Harbinger measured application throughput and latency using Microsoft Visual Studio Team System (VSTS 2008). Over the next phase, Harbinger used ANT's profiler, Microsoft CLR, SQL profiler and memory profiler to profile code and monitor external resources utilization. Harbinger analyzed results and recorded parameters such as page response times and memory utilization. Progressive check points were used by the customer and Harbinger to optimize code and increase load in steps.

Since the software was delivered as a service, testing on various hardware configurations like single machine, load-balanced server farms and controller-agent configurations was critical. Harbinger used VSTS in these configurations to ensure high uptime and robustness.  
 

• Microsoft Visual Studio Team System 2008 (VSTS)
• ANT profiler
• Microsoft CLR
• SQL profiler
• Memory profiler

Benefits

The product went from a modest capacity of couple of hundred thousand users to millions. The number of concurrent users that the system supports went from a mere 10 to 1000. Step-load user capacity, a more realistic measure of actual load on the system went up to 3000 users. The product can now on-board up to 100,000 users at a time in half the time that it used to on-board 100 users, enabling the company to strike bigger deals and executing much larger contracts. Above all, the company is very confident today about the capacity and performance of its product, putting it on a significantly higher growth trajectory.