Performance Testing Lowdown

Craig Kilshaw is a Test Engineer at Storm ID. Below, he takes us through what performance testing is, exploring specific performance testing approaches that we use for websites and web applications.

This is the description that gets returned on Google if you search for performance testing:

“”Performance Testing is a software testing process used for testing the speed, response time, stability, reliability, scalability and resource usage of a software application under particular workload. The main purpose of performance testing is to identify and eliminate the performance bottlenecks in the software application. It is a subset of performance engineering also known as “Perf Testing””.

While this is accurate, I feel it is a very broad statement giving little information with no specifics. I am going to drill down a bit further into what performance testing entails, what tests we can or should run, why we should run them and what we expect to find out after testing is completed.

Due to the complexity of modern software, there are lots of areas that could be looked at in terms of performance. We could test the infrastructure looking purely at database connections, we could test the API end points focusing on service response times, we could drive a site through the AI and see how the application will perform once it is out in the world with a real user load, or we could use a combination of them all to give a complete picture of performance at all levels.

Performance testing can be considered as a form of automated testing since it involves the same practices of creating test scripts, then using a tool to drive those scripts without any additional interaction from the tester – although the intended outcomes for these are different since automated tests are generally used to verify the user journey, whereas performance tests are used to verify the user experience.

Performance testing falls into the category of non-functional testing, along with other important testing that has a direct impact on the ongoing usability of a site or application, such as accessibility testing and security testing. More so than other forms of testing, performance is driven by targets and non-functional requirements established early in the project. The idea is to simulate usage of the service or site we are testing, so knowing what the expected user load will be or what the expected high traffic times look like, allows us to create specific scenarios relevant to that service. The more we know the more detailed those scenarios can become.

The number of variables that could affect performance results means that it needs to ideally be run in lab conditions to confirm you are comparing one set of relevant results against another. For instance, a database may perform slower as it fills up, so if we run a test against a database with ten entries, and then we run the same test against the same database when it has ten thousand entries, then those results cannot always be expected to return the same performance on the front end due to additional time required from the database. This mean knowing exactly what we are aiming to confirm while testing is important, so we can, if possible, set up the environment correctly then reset it back to that state for another round of testing when we are looking to track performance improvements or degradation.

This is really the goal of performance testing, since knowing how well your application works at launch is useful, but knowing if it is still operating with the same performance after six months or a year of regular or simulated usage will help drive decisions about what could be changed or streamlined in future development.

Here is list of common types of performance testing techniques. I will discuss them as though we are testing API end points, although these could also be applicable to both front end and infrastructure testing as well.

Load Testing

This is what most people would consider to be standard performance testing and should be applied later in a project when most of the design, development and content population has already been completed and tested. It is a useful way to find any potential bottlenecks, which are any point in the journey that consistently slows down traffic.

Endurance Testing

Also referred to as soak testing and could be considered a component of load testing. This is used to measure how well an application performs under load over time. Scripts are created to simulate a load on the application over large periods of time – days or even weeks.

For both these test types, we would start by creating some scripts of regular usage of the site. These are created to simulate people passing through and using the application under test. They could log in, upload/download files, purchase things from a shop, etc. Then, using our performance tool, we create a virtual user load that will run these scripts randomly at the same time, and we will record various outcomes from that test to report back.

This testing is generally used to verify metrics such as

• Throughput – The number of times per second a service can respond
• Maximum response time
• Failure Rate

Stress Testing

Stress testing is used in the same way that load testing is, in that it is used to simulate expected usage of the application. The difference being that it is done using a much larger number of users than is expected for the application under test. The idea is to find the breaking point of whatever is being tested to find what breaks first, and if improvements can be made, to allow for the unexpected high volume of traffic.

Spike Testing

Spike testing falls under stress testing and is used for looking at high traffic services which do not see sustained usage. It is generally used for testing something like a ticket website that expects lots of traffic when new event tickets are released, or a utility company contact page for reporting power outages in a storm.

A script is created for the specific service to test rapidly increasing the load, for example going from 0 to 1000 users then back again in 1 second. The pattern is then repeated. It allows us to test if the service can handle the load, and lets us find out when performance issues start being seen.

This testing is generally used to verify metrics such as

• Maximum user load before critical failure
• Time for application to recover after failing
• Maximum response time
• Failure rate

Hopefully, this provides a little bit more context into what performance testing is and what value it can bring to website and application development. There’s no one size fits all approach for this. Some of our smaller websites and applications may only need some load testing applied to the verify front-end changes, while others might only need some spike testing applied to an updated login service. All our larger websites and applications will certainly see a benefit from us throwing everything we have at it. But knowing what we need to test and what we are testing for, is the most important thing to allow us to perform targeted testing specific to that product.