This week we introduced the the biggest change yet to template editing on Roboswarm. With improved support for complex actions (PATCH, PUT, DELETE, POST) and new support for sequenced requests, Roboswarm is more powerful than ever!
The new template editor brings a lot of advanced functionality to our WordPress load testing tool, while still maintaining ease of use for people not comfortable with load testing.
We now support more HTTP methods
GET
PATCH
POST
PUT
DELETE
Custom HTTP headers
Custom HTTP body
Regular form encoded HTTP bodies as well as JSON encoded are supported.
For WordPress specifically, you can now authenticate users by uploading a CSV with a combination of usernames and passwords.
In addition to the template editor changes, load test templates created with the new editor can now choose between two user behavior types:
Even spread – The load test users spread out evenly across the entire template
Sequence – The load test users step through the routes in the template in order.
Even spread is the default behavior for RoboSwarm, but sequence style tests are extremely useful for modeling specific user flows such as authentication.
When hosting a WooCommerce store performance is critical. The relationship between time to checkout and dollars earned is direct and measurable. In this article we’re going to take a look at WooCommerce DigitalOcean performance and see if we can draw any conclusions. Our hope is that with the load testing runs we’ve done on RoboSwarm you’ll be able to make an informed decision around hosting your WooCommerce store.
Performance and load testing WooCommerce is an interesting problem. You need to have session handling between requests and be able to submit CSRF tokens. RoboSwarm is able to do proper WooCommerce load testing with adding items to your cart, viewing your cart, and checking out. The metrics we care about are:
WooCommerce Checkout Performance – Your customers checking out is where money is made. Performance of checkout is arguably the most important measure we can make.
Overall WooCommerce Performance – While checkouts are important, your store’s general performance also plays a big part in getting customers to purchase items.
How are we testing?
In order to get a good understanding of WooCommerce performance on DigitalOcean, each virtual user goes through the following scenario on repeat:
Go to the store main page.
Go to product 1, add to cart.
Go to product 2, add to cart.
Go to product 3, add to cart.
Go to cart.
Checkout
Load checkout success page.
While more direct than most users, its fairly representative of how customers shop (add a few items, view the cart, checkout). As far as the tests we ran, each test was 200 concurrent virtual users, for 45 minutes, with traffic being generated in DigitalOcean’s San Francisco 2 data center and targeting their New York City 3 data center. As far as what DigitalOcean machine configurations we tested:
W3 Total Cache backed by Memcached. All pages except cart and checkout are full-page cached.
WooCommerce DigitalOcean Checkout Performance
The table below represents the sum of the data from all of our test runs. Before we go into each category in depth, it’s worth calling out some interesting data points.
As checkout throughput increased, the response time distribution and average/median response times actually increased. Meaning raw checkouts per second was higher but response time was slower.
WooCommerce is really slow if you don’t have at least 2 vCPUs.
DigitalOcean’s CPU optimized droplets didn’t provide much in the way of performance increases over their standard droplets.
Checkouts Per Second
Avg Response Time
Median Response Time
P50
P90
P99
Standard: 1vCPU, 1GB RAM
0.175
16118
16000
16000
25000
27000
Standard: 1vCPU, 1GB RAM [CACHED]
0.588
17860
17000
17000
27000
33000
Standard: 2vCPU, 2GB RAM
1.175
6200
5802
6200
8900
14000
Standard: 2vCPU, 2GB RAM [CACHED]
1.58878
8200
7998
8200
11000
12000
Standard: 4vCPU, 8GB RAM
2.676
3500
3429
3500
4900
6500
Standard: 4vCPU, 8GB RAM [CACHED]
3.173
4700
4689
4700
5900
6800
Standard: 8vCPU, 16GB RAM
4.012
4100
3986
4100
5000
5700
Standard: 8vCPU, 16GB RAM [CACHED]
6.642
2614
2600
2600
3400
4300
CPU Optimized: 2vCPU, 4GB
0.747
8662
10000
10000
14000
15000
CPU Optimized: 2vCPU, 4GB [CACHED]
1.386
9411
9200
9300
13000
14000
CPU Optimized: 4vCPU, 8GB
2.543
4145
4500
4500
5200
5800
CPU Optimized: 4vCPU, 8GB [CACHED]
3.32
4612
4600
4600
6100
7500
CPU Optimized: 8vCPU, 16GB
4.654
3508
3600
3085
4500
5200
CPU Optimized: 8vCPU, 16GB [CACHED]
5.19
4429
4200
4200
6900
8900
All times in milliseconds
Checkouts per Second
Checkouts per Second
The checkouts per second metric for WooCommerce performance is likely the most important metric that we have. You can see that in un-cached scenarios WooCommerce and WordPress do pretty terrible. Once your throw a bit of caching in though things become a little more reasonable from a throughput perspective. As an example, if you were processing $50 orders, the table below shows you the maximum number of dollars your could make in a given minute, hour, or day.
MINUTE
HOUR
DAY
1vCPU/1GB [CACHED]
$1,764
$105,840
$2,540,160
2vCPU/2GB [CACHED]
$4,764
$285,840
$6,860,160
4vCPU/8GB [CACHED]
$9,519
$571,140
$13,707,360
8vCPU/16GB [CACHED]
$19,926
$1,195,560
$28,693,440
Unrealistic
Being realistic, this is a very unlikely scenario. But it does illustrate a point: The amount of money you pay for your WooCommerce hosting can directly impact the bottom line of your business. Are you going to be processing 6.642 orders per second? Probably not. But traffic might spike at some point, and you don’t want to turn away anyone because your performance is terrible.
Checkout Average Response Time
Average Response Time
In the scenario above, having a lower number is better. As you can see the cheapest DigitalOcean droplet has really bad performance when under heavy load. A checkout taking 17s on average is really not a great experience. Things improve quite a bit as you go into the more expensive tier droplets, but I personally wouldn’t be happy until I got below 5s.
An interesting point in the data above is that in most cases the cached performance is worse than the un-cached performance. Why though? Because in the cached scenario the throughput is higher.
Response Time Distribution
The final metric for checkouts we’re going to look at is response time distribution. What does that mean? It basically means that we’re going to look at 50%, 90%, and 99% of requests and see how fast they finish. For example if the 90% distribution is 2000ms, that means that 90% of requests finished in 2000ms or under.
Response Time Distribution: 1vCPU, 1GB RAM
Image 1 of 7
If you go through the gallery above and focus on the 50% and 99% numbers, you can see that there is quite a wide range of experiences your customers will have. Ideally we want to see the 50% and 99% number a lot closer together and low, because this indicates that the response time for all customers is similar.
Now that we’ve looked specifically at checkout performance, lets take a broader look at the overall performance. For this section, we’re going to focus exclusively on the 4 vCPU / 8GB RAM cached variant of the test.
Overall store performance wasn’t too bad with caching enabled. The cache was able to handle the vast majority of the pages, with checkout and cart pages being the exception.
Requests / Failures per Second
You can see that as the user load went up the request throughput managed to stay roughly the same, fielding ~40 req/s on average. We didn’t see any errors in this scenario either.
Average & Median Response Time
The average and median response time under full load was around 3.9s. Both times mirrored each other closely, which indicates that we didn’t have too many outliers skewing the average time. In general this is a good thing because we want performance to be consistent and we don’t want outliers in the high range of response times.
Response time distribution
The overall response time distribution wasn’t too bad. 50% of requests finished in under 4s, and 99% finished in under 8s. If this were a normal “read only” WordPress site these numbers would be awful, but given the amount of requests that must be un-cached, it isn’t too bad at all.
Conclusions
With WordPress it is often the case that you can get away with a cheap $5/month host & some caching. With WordPress + WooCommerce that isn’t true. Because of the routes that aren’t cache-able you need some more horsepower to give your customers a decent experience. Our opinion is that if you are hosting a WooCommerce store on DigitalOcean, be sure to run it on a Droplet with at least 2 vCPUs. Whether you go further than that depends on the typical traffic levels on your store.