One of the ways that RoboSwarm keeps it’s costs down is by spinning up infrastructure to run load tests on-demand. While great for our customers and us, it does lead to some annoying behavior when it comes to startup times. Very recently we released a feature to help customers understand when their load test will start.
To get the “~121 seconds” and descending number we do a little bit of fancy SQL to determine the average time to create a load testing cluster.
EPOCH FROM (
ready_at - created_at
) as avg_time
ready_at IS NOT NULL
destroyed_at IS NOT NULL
AND id < ?
AND id >= (? - 10)
The above code subtracts the created_at datetime from the ready_at datetime, generates the time in seconds, and averages that across the previous 10 load tests from the point where this current load test was created. So why not averaging the last 50 or 100 test startup times? Mainly because the time to create tests has trended down in the past year and we want to give you the most accurate representation.
Generating this with a single query to PostgreSQL is fantastic and removes a lot of overhead and complication from the code base.
This week we introduced the the biggest change yet to template editing on Roboswarm. With improved support for complex actions (PATCH, PUT, DELETE, POST) and new support for sequenced requests, Roboswarm is more powerful than ever!
The new template editor brings a lot of advanced functionality to our WordPress load testing tool, while still maintaining ease of use for people not comfortable with load testing.
We now support more HTTP methods
Custom HTTP headers
Custom HTTP body
Regular form encoded HTTP bodies as well as JSON encoded are supported.
For WordPress specifically, you can now authenticate users by uploading a CSV with a combination of usernames and passwords.
In addition to the template editor changes, load test templates created with the new editor can now choose between two user behavior types:
Even spread – The load test users spread out evenly across the entire template
Sequence – The load test users step through the routes in the template in order.
Even spread is the default behavior for RoboSwarm, but sequence style tests are extremely useful for modeling specific user flows such as authentication.
When hosting a WooCommerce store performance is critical. The relationship between time to checkout and dollars earned is direct and measurable. In this article we’re going to take a look at WooCommerce DigitalOcean performance and see if we can draw any conclusions. Our hope is that with the load testing runs we’ve done on RoboSwarm you’ll be able to make an informed decision around hosting your WooCommerce store.
Performance and load testing WooCommerce is an interesting problem. You need to have session handling between requests and be able to submit CSRF tokens. RoboSwarm is able to do proper WooCommerce load testing with adding items to your cart, viewing your cart, and checking out. The metrics we care about are:
WooCommerce Checkout Performance – Your customers checking out is where money is made. Performance of checkout is arguably the most important measure we can make.
Overall WooCommerce Performance – While checkouts are important, your store’s general performance also plays a big part in getting customers to purchase items.
How are we testing?
In order to get a good understanding of WooCommerce performance on DigitalOcean, each virtual user goes through the following scenario on repeat:
Go to the store main page.
Go to product 1, add to cart.
Go to product 2, add to cart.
Go to product 3, add to cart.
Go to cart.
Load checkout success page.
While more direct than most users, its fairly representative of how customers shop (add a few items, view the cart, checkout). As far as the tests we ran, each test was 200 concurrent virtual users, for 45 minutes, with traffic being generated in DigitalOcean’s San Francisco 2 data center and targeting their New York City 3 data center. As far as what DigitalOcean machine configurations we tested:
W3 Total Cache backed by Memcached. All pages except cart and checkout are full-page cached.
WooCommerce DigitalOcean Checkout Performance
The table below represents the sum of the data from all of our test runs. Before we go into each category in depth, it’s worth calling out some interesting data points.
As checkout throughput increased, the response time distribution and average/median response times actually increased. Meaning raw checkouts per second was higher but response time was slower.
WooCommerce is really slow if you don’t have at least 2 vCPUs.
DigitalOcean’s CPU optimized droplets didn’t provide much in the way of performance increases over their standard droplets.
Checkouts Per Second
Avg Response Time
Median Response Time
Standard: 1vCPU, 1GB RAM
Standard: 1vCPU, 1GB RAM [CACHED]
Standard: 2vCPU, 2GB RAM
Standard: 2vCPU, 2GB RAM [CACHED]
Standard: 4vCPU, 8GB RAM
Standard: 4vCPU, 8GB RAM [CACHED]
Standard: 8vCPU, 16GB RAM
Standard: 8vCPU, 16GB RAM [CACHED]
CPU Optimized: 2vCPU, 4GB
CPU Optimized: 2vCPU, 4GB [CACHED]
CPU Optimized: 4vCPU, 8GB
CPU Optimized: 4vCPU, 8GB [CACHED]
CPU Optimized: 8vCPU, 16GB
CPU Optimized: 8vCPU, 16GB [CACHED]
All times in milliseconds
Checkouts per Second
The checkouts per second metric for WooCommerce performance is likely the most important metric that we have. You can see that in un-cached scenarios WooCommerce and WordPress do pretty terrible. Once your throw a bit of caching in though things become a little more reasonable from a throughput perspective. As an example, if you were processing $50 orders, the table below shows you the maximum number of dollars your could make in a given minute, hour, or day.
Being realistic, this is a very unlikely scenario. But it does illustrate a point: The amount of money you pay for your WooCommerce hosting can directly impact the bottom line of your business. Are you going to be processing 6.642 orders per second? Probably not. But traffic might spike at some point, and you don’t want to turn away anyone because your performance is terrible.
Checkout Average Response Time
In the scenario above, having a lower number is better. As you can see the cheapest DigitalOcean droplet has really bad performance when under heavy load. A checkout taking 17s on average is really not a great experience. Things improve quite a bit as you go into the more expensive tier droplets, but I personally wouldn’t be happy until I got below 5s.
An interesting point in the data above is that in most cases the cached performance is worse than the un-cached performance. Why though? Because in the cached scenario the throughput is higher.
Response Time Distribution
The final metric for checkouts we’re going to look at is response time distribution. What does that mean? It basically means that we’re going to look at 50%, 90%, and 99% of requests and see how fast they finish. For example if the 90% distribution is 2000ms, that means that 90% of requests finished in 2000ms or under.
Response Time Distribution: 1vCPU, 1GB RAM
Image 1 of 7
If you go through the gallery above and focus on the 50% and 99% numbers, you can see that there is quite a wide range of experiences your customers will have. Ideally we want to see the 50% and 99% number a lot closer together and low, because this indicates that the response time for all customers is similar.
Now that we’ve looked specifically at checkout performance, lets take a broader look at the overall performance. For this section, we’re going to focus exclusively on the 4 vCPU / 8GB RAM cached variant of the test.
Overall store performance wasn’t too bad with caching enabled. The cache was able to handle the vast majority of the pages, with checkout and cart pages being the exception.
You can see that as the user load went up the request throughput managed to stay roughly the same, fielding ~40 req/s on average. We didn’t see any errors in this scenario either.
The average and median response time under full load was around 3.9s. Both times mirrored each other closely, which indicates that we didn’t have too many outliers skewing the average time. In general this is a good thing because we want performance to be consistent and we don’t want outliers in the high range of response times.
The overall response time distribution wasn’t too bad. 50% of requests finished in under 4s, and 99% finished in under 8s. If this were a normal “read only” WordPress site these numbers would be awful, but given the amount of requests that must be un-cached, it isn’t too bad at all.
With WordPress it is often the case that you can get away with a cheap $5/month host & some caching. With WordPress + WooCommerce that isn’t true. Because of the routes that aren’t cache-able you need some more horsepower to give your customers a decent experience. Our opinion is that if you are hosting a WooCommerce store on DigitalOcean, be sure to run it on a Droplet with at least 2 vCPUs. Whether you go further than that depends on the typical traffic levels on your store.
WooCommerce is one of the biggest players in the e-commerce ecosystem, but being built on top of WordPress means that it’s performance can sometimes be underwhelming. One way to increase the performance of WooCommerce is to also run a caching plugin like W3 Total Cache.
W3 Total Cache can be a bit overwhelming to newcomers, and the extra configuration required by W3TC is just another layer of complexity. The good news is that if you have a bit of knowledge, configuring W3 Total Cache for WooCommerce isn’t too hard.
The first thing you need to do once you’ve installed W3 Total Cache is enable all the caches. This includes:
Going in to what each of these does is beyond the scope of this article, but go ahead an enable them all. You are unlikely to run in to a situation where these being enable will cause you trouble.
This is the part of the exercise that is specific to WooCommerce. To make W3 Total Cache work with WooCommerce, you need to make sure that the dynamic pages of WooCommerce are added to the exception list. To add pages to the exception list go to: Performance -> Page Cache. After that scroll to “Never Cache the Following Pages” and add:
cart – This is the page where your customers go to see what’s in their cart. This should remain dynamic and not be cached.
checkout – This is the page where your customers go to check out. This should also stay dynamic.
That’s all there is to it! You’re WooCommerce site will not perform much better.