The odd thing about web load testing is that there are a dozen different reasons that a client might be interested in doing it. And there are a dozen different roles that a tester might take on in order to be an effective member of the team. This course is about explaining the motivations of both sides, and the basic methodology that we as testers can use to accomplish our clients’ goals.
Clients like to break things! Sometimes. Other times, clients want to put an extraordinary amount of stress on their system until it slows to a crawl. Or they want to find the limits of their software, or the limits of standardized hardware running their software. Or they want to see how dependable their system is under constant ‘normal’ usage. Each client may focus on a different reason to do the load testing, but the methods for the testing itself may be similar. Let’s look at some of the reasons in detail:
1) The client is looking for a breaking point in their software:
This is also called ‘software stress testing’. The point of this kind of test is to see where the software starts to break down under extreme load. Rapid memory leaks and faults, buffer mishandling, threading issues, and the like can all start to happen with alarming frequency when a piece of software starts to reach its limits. This can often be done in conjunction with hardware stress testing.
2) The client is looking for the breaking point of the standardized hardware running their software:
This is also called ‘hardware stress testing’. The point of this kind of test is to see where a benchmarked set of hardware starts to break down under extreme load when running the client’s software. Running low on memory, processing power, or buffers are all possible. If limitations in the software are reached prior to reaching the limits of the hardware, this can easily become a software stress test.
3) The client wants to make sure that their system is robust under long term normal usage:
This is also called ‘soak testing’. This is a simulation that often takes place over the course of days or weeks that tests for performance degradation caused by things like slow memory leaks or even something as simple as hardware fatigue or subpar hosting conditions.
4) The client wants to see if their software will properly scale under sudden increased loads:
This is also called ‘spike testing.’ The ability of an application to shift gears and go from handling a moderate number of users efficiently, to handling a large number of users adequately, is very important. If a client’s company makes the front page of a news website or an aggregate like Digg or Slashdot, its normal load can spike quite heavily.
5) The client wants to compare relative performance based on sets of server parameters:
This is also called ‘configuration testing.’ If a client is unsure of what hardware and software options are best for their application, they can run a series of tests with the exact same user load, and monitor the relative performance in each scenario.
6) The client wants to test end user functionality on a broad spectrum of hardware and software:
This is also called ‘platform testing’. This allows a client to test their software’s front end under a wide variety of hardware and software configurations. This helps them determine their software’s requirements, which they can publish as their suggested minimum configuration.
In response to the client’s needs, a load test plan is crafted by the lead tester. They will determine what kind of testing is required from the list below, and assign the appropriate loads to the testers. The three main kinds of load testing for web applications are:
A) Automated Virtual Users:
This is a non functional test that simply emulates a bunch of users focused on using a certain section of the web site or application. Automated virtual users are created by injecting their actions directly into hypertext streams, and sending them to the server. This is suitable for emulating dozens, or hundreds of users. No further action is necessary from the tester unless instructed mid-test, other than logging and monitoring.
B) Front End Automation:
Often used in regression testing, front end automation involves scripting the actual mouse movements, mouse clicks, and text entry of one or more virtual users. This is record and playback automation, and is not suitable for emulating more than a few users. This incorporates a level of functional testing to the load test, important if there is some amount of importance in testing the front end stability and predictability of the GUI.
C) First Person Participation:
Used when personal observation is required, and subjective qualities need to be rated. The tester might be executing automated tests, and still participate in first person testing. The tester will be able to more easily rate things like sound and video quality, and perceived responsiveness.
We’ll be discussing specific tools for these testing types later in the course. For the moment, let’s talk about what NOT to do during a load test:
- Don’t spam the live test with details of your functional bugs unless the client has requested that you do so.Simply put, a load test is a highly controlled bombardment of hardware and software, and everyone needs to be concentrating on monitoring the status of the server. If the client has asked you to log functional bugs, feel free. On your own time.
- If three or four people have answered a test coordinator’s question, don’t add your answer unless it is a different answer or you’ve specifically been instructed to answer! In other words, if someone asks ‘Can you hear me now?’ and a bunch of people have answered ‘Yes’… unless you didn’t hear them for some reason, we don’t need any more confirmation. Don’t add to the spam. It’s very distracting during a live test.
- Don’t add more to the load than you’ve been instructed to do! If the test coordinator needs more virtual users, believe me, he’ll ask you or one of the other testers. Don’t take it upon yourself to add more virtual users without permission. Load testing is about exact numbers and well executed plans.
- If you have questions, use the proper channels and make sure it isn’t answered in the test instructions. This implies, of course, that you should have read the load test instructions in full well before the test started. If you have an urgent question, then address it to the lead tester rather than directly to the client.
That covers the general mentalities involved in the various types of web based load testing.