Setup and Run your distributed JMeter load test in just 1 minute
When starting to load test a target system with Apache JMeter, we first write the JMeter tests with GUI in a local machine and and execute and debug the tests. Local execution is only okay for you to get test scripts working. If the team is in a hurry to get load test results, what we used to do is setup an VM and move the JMeter scripts in to the VM and execute the tests.
But what if need to run the tests couple of times? What if load tests results revealed issues in the performance? and after fixing the issues also you will have to rerun the tests on VM. And may be just one VM won’t be enough to run if you have to generate a huge load for your tests, then you will need to do a JMeter distributed setup with multiple workers.
What if there’s a way to setup run your load tests on a distributed setup with less time than you’ll take to setup a VM? What if you could integrate these tests in to the CI/CD pipeline? sounds good right? And that’s exactly what Azure has introduced with their Azure load testing preview.
Azure Load Testing Preview is a fully managed load-testing service that enables you to generate high-scale load. Ultimately reducin time you spent provisioning and maintaining virtual machines and increase the time you spend load testing.
Prerequisites
- An Azure account with an active subscription. If you don’t have an Azure subscription, create a free account before you begin.
Getting started with Azure Load Testing
- Navigate to https://azure.microsoft.com/en-us/services/load-testing/
- You’ll need to create load Testing resource
- After creating the required resource, you will be navigated to the following page
To implement your first test, click on the Tests button on left panel.
The above page is where you can see the list of tests. As you have not written any tests yet, your list will be empty.
Create your first load test
There are two ways to create load tests in Azure load tests,
- Create a quick test by using a web application URL.
2. Create a test by uploading a JMeter test script (JMX).
This is the the one you should be working with. As you already know, for this method you should have a JMeter test script. You can use the following JMeter script I have created for a simple sample test. Just create a file in your local directory with name sampleloadtest.jmx and paste the following code.
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.4.2">
<hashTree>
<TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
<stringProp name="TestPlan.comments"></stringProp>
<boolProp name="TestPlan.functional_mode">false</boolProp>
<boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
<boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
<elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
<collectionProp name="Arguments.arguments">
<elementProp name="concurrentUsers" elementType="Argument">
<stringProp name="Argument.name">concurrentUsers</stringProp>
<stringProp name="Argument.value">${__BeanShell( System.getenv("concurrentUsers") )}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="rampUpTime" elementType="Argument">
<stringProp name="Argument.name">rampUpTime</stringProp>
<stringProp name="Argument.value">${__BeanShell( System.getenv("rampUpTime") )}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
</collectionProp>
</elementProp>
<stringProp name="TestPlan.user_define_classpath"></stringProp>
</TestPlan>
<hashTree>
<ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
<stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
<elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
<boolProp name="LoopController.continue_forever">false</boolProp>
<stringProp name="LoopController.loops">1</stringProp>
</elementProp>
<stringProp name="ThreadGroup.num_threads">${concurrentUsers}</stringProp>
<stringProp name="ThreadGroup.ramp_time">${rampUpTime}</stringProp>
<boolProp name="ThreadGroup.scheduler">false</boolProp>
<stringProp name="ThreadGroup.duration"></stringProp>
<stringProp name="ThreadGroup.delay"></stringProp>
<boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
</ThreadGroup>
<hashTree>
<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="jsonplaceholder Request" enabled="true">
<elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
<collectionProp name="Arguments.arguments"/>
</elementProp>
<stringProp name="HTTPSampler.domain">jsonplaceholder.typicode.com</stringProp>
<stringProp name="HTTPSampler.port"></stringProp>
<stringProp name="HTTPSampler.protocol">https</stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path">/posts</stringProp>
<stringProp name="HTTPSampler.method">GET</stringProp>
<boolProp name="HTTPSampler.follow_redirects">true</boolProp>
<boolProp name="HTTPSampler.auto_redirects">false</boolProp>
<boolProp name="HTTPSampler.use_keepalive">true</boolProp>
<boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
<stringProp name="HTTPSampler.embedded_url_re"></stringProp>
<stringProp name="HTTPSampler.connect_timeout"></stringProp>
<stringProp name="HTTPSampler.response_timeout"></stringProp>
</HTTPSamplerProxy>
<hashTree/>
<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="facebook Request" enabled="true">
<elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
<collectionProp name="Arguments.arguments"/>
</elementProp>
<stringProp name="HTTPSampler.domain">www.facebook.com</stringProp>
<stringProp name="HTTPSampler.port"></stringProp>
<stringProp name="HTTPSampler.protocol">https</stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path">/login</stringProp>
<stringProp name="HTTPSampler.method">GET</stringProp>
<boolProp name="HTTPSampler.follow_redirects">true</boolProp>
<boolProp name="HTTPSampler.auto_redirects">false</boolProp>
<boolProp name="HTTPSampler.use_keepalive">true</boolProp>
<boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
<stringProp name="HTTPSampler.embedded_url_re"></stringProp>
<stringProp name="HTTPSampler.connect_timeout"></stringProp>
<stringProp name="HTTPSampler.response_timeout"></stringProp>
</HTTPSamplerProxy>
<hashTree/>
<ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="true">
<boolProp name="ResultCollector.error_logging">false</boolProp>
<objProp>
<name>saveConfig</name>
<value class="SampleSaveConfiguration">
<time>true</time>
<latency>true</latency>
<timestamp>true</timestamp>
<success>true</success>
<label>true</label>
<code>true</code>
<message>true</message>
<threadName>true</threadName>
<dataType>true</dataType>
<encoding>false</encoding>
<assertions>true</assertions>
<subresults>true</subresults>
<responseData>false</responseData>
<samplerData>false</samplerData>
<xml>false</xml>
<fieldNames>true</fieldNames>
<responseHeaders>false</responseHeaders>
<requestHeaders>false</requestHeaders>
<responseDataOnError>false</responseDataOnError>
<saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage>
<assertionsResultsToSave>0</assertionsResultsToSave>
<bytes>true</bytes>
<sentBytes>true</sentBytes>
<url>true</url>
<threadCounts>true</threadCounts>
<idleTime>true</idleTime>
<connectTime>true</connectTime>
</value>
</objProp>
<stringProp name="filename"></stringProp>
</ResultCollector>
<hashTree/>
</hashTree>
</hashTree>
</hashTree>
</jmeterTestPlan>If you open the above test with your locally installed JMeter app, it will look as follows.
If you look in user defined variables of the Test plan (figure 1), you can see I have parameterized them as system variables and used the values inside the thread group for Thread properties. Later we will be setting up our azure tests to pass values in to those parameters.
Upload test plan
Ok, now we’re ready to start creating our first test. Just go ahead and click on Create button in the test plans page and select Upload a JMeter script from the drop down.
Enter a name and description to your test plan.
Click next button.
Select the .jmx we created from your local directory and click Upload.
once uploaded, click next button.
Configure parameters
Now you are in the Parameters page. You can add parameters as Environment variables or Secrets which are sensitive parameters (user credentials, etc.) Those secrets will be stored in Azure Key Vault and you can read more about that here.
As we have only parameterized the no. of concurrent users and ramp up period in our tests, we can use environment variables to pass them. Add the parameters as follows,
The values we enter for the parameters will be used as default values when executing the tests, but you can also customize them later.
Click next.
Configure test engines
Test engine instances are responsible for executing a test plan. Each test engine executes the Apache JMeter script. The test engine instances run in parallel. They allow you to define how you want to scale out the load test execution for your application. Azure recommend that you keep the number of threads below a maximum of 250 in the test plan.
For example, to simulate 1,000 threads (or virtual users), set the number of threads in the Apache JMeter script to 250. Then configure the test with four test engine instances (that is, 4 x 250 threads). As we have already parameterized the Threads value in the test plan, we can set the value when running the tests.
As we are not going to put a huge load in the example, I’m going to keep the instances to just 1. Click next button.
Test criteria
In Test criteria you can define when to fail a test.
We have following 2 metrics right now,
- Response_time_ms
- Error
After selecting the Metric you can select the Aggregate function (average, percentage), and condition as Greater than and the Threshold number.
for example, You can define criteria to fail the test when Error percentage is grater than 1%. Also you can define multiple criteria up to 10.
Monitoring
With this feature you can monitor server-side application metrics in real time for Azure-hosted applications when running a load tests. It’s a very useful and powerful feature for identifying performance issues and bottlenecks with the resources. You no longer need to ask the area owners to check logs and monitor the resources and rerun the tests when there is an issue.
To monitor resource metrics for an Azure-hosted application, you need to specify the list of Azure application components in your load test. Azure Load Testing automatically captures a set of relevant resource metrics for each selected component. When your load test finishes, you can view the server-side metrics in the dashboard.
Review + create
That’s it! Now we are done with creating the load test plan. You can click Create button to finish things.
Your test will auto trigger for the first time and if everything goes right you will be able to see the results as follows.
Download reports
If you need more information about the requests specially when there are errors, you can download the results by clicking on the Download button and selecting Results.
A file will downloaded in .csv format. For you to read the data in this file, you can open the report with a JMeter listener. I prefer using “View Results Tree”. Just create a new test plan in JMeter and add Listener View Results Tree. Go to the Listener and browse and import the downloaded .csv file.
Running tests again
To run the tests again click on your test plan name.
You will be navigated a page with history of the tests runs of the plan.
Click on Run button. A panel will pop up to setup the test. Here also you can edit the parameter values you have set.
Test failures
When there are test failures, the dashboard will look as follows. You can dig deeper in to the results by download and opening the report .csv with JMeter.
Other cool features in Azure Load Testing Preview
Move between Regions
Azure Load Testing resources are region-specific and can’t be moved across regions automatically. You can use an Azure Resource Manager template (ARM template) to export the existing configuration of your Load Testing resource instead. Then, stage the resource in another region and create the tests in the new resource. Read more about how to do this.
Server side Monitoring
As I mentioned earlier you can setup server side monitoring for your tests and create a dashboard as follows,
Identify performance regressions by comparing test runs
You can compare load test runs for the following scenarios:
- Identify performance regressions between application builds or configurations. You could run a load test at each development sprint to ensure that the previous sprint didn’t introduce performance issues.
- Identify which application component is responsible for a performance problem (root cause analysis). For example, an application redesign might result in slower application response times. Comparing load test runs might reveal that the root cause was a lack of database resources.
Continuous regression testing
You can set up an Azure Pipelines CI/CD workflow or GitHub Actions CI/CD workflow to deploy your application and trigger a load test using the Azure Load Testing action.
Conclusion :
Azure Load Testing is currently in preview and very new to the users. There are bugs and required features yet to come but still this offers a lot of powerful features that will make your life easier. Most of the projects do performance tests at later stage of the development just before the releases, so it is very challenging for QAs and developers to write the tests, prepare the distributed environments, Execute the tests in a short amount of time. Azure Load Testing take care of more than half of that effort and helps you identifying the performance bottlenecks too.
Azure load testing product documentation : https://docs.microsoft.com/en-gb/azure/load-testing/
Report issues here : https://github.com/microsoft/azure-load-testing/issues
