One of the key reasons why people buy Rainforest is for the benefit of a crowd-powered QA testing solution. So how do you balance speed against a crowd of folks who have to take things step-by-step and be as thorough as they can be?
We’re often asked, “How can I get my tests to run faster?”
When time is of the essence, here are some tips and tricks to speed things up a bit, whether you’re running one or multiple tests:
QA strategy tips
- Run on fewer platforms.
If speed is paramount, instead of running a test on four browsers, narrow it down two platforms that are crucial.
- Make the tests short and modular: under 25 steps.
The shorter and more modular a test is, the faster and more accurate the test results will be. Each test should cover one feature or functionality, we recommend avoiding long end-to-end tests. Whenever you add a step, not only does this increase execution time, but it also adds a potential new point of failure.
- Break steps up, and don’t cram too much in them.
Our testers have to read, process, execute, and remember what you’ve asked them to do. A quick succession of buttons to click or fields to enter is fine. Asking testers to complete a 30-field form in one step is not. Testers want to be accurate, which adds to the time it takes them to complete your step.
- Combine steps where it makes sense to get to the point of the test.
Think critically about how granular each step is, and what you’re really checking for. It also makes sense to structure your question so that it clearly links to the actions described in the step.
- Try running on a popular browser.
Runs using uncommon VM types can also be slower, as they are not in high demand. If it’s not integral to your testing strategy, try browsers like chrome_1440_900.
- Check if there’s a concurrent tester limit on your account.
This is the max number of testers that can access your product at the same time. It’s used to throttle traffic on limited-capacity servers, and may slow down run times as we limit the number of testers that can test at a time. You can find it in Settings > Global Settings
- Take tester feedback into account.
When they pass a step, testers also have the option to leave comments about items they noticed on your site or confusing instructions, to help you refactor instructions. If they don’t understand what you want, they will dwell on that step for a long time.
- Examine tester videos to see how much time is spent on each step.
Our videos have markers for step completion and can be sped up to 5x speed if needed. If the tester spent 10 minutes on a step, perhaps the instructions aren't clear and should be evaluated or broken up.
- Preview the test when writing or refactoring:
Can you go through the test yourself in a reasonable amount of time? The testers will experience it just like you, except they are less familiar with your application and it may take them longer.
- Add screenshots or GIFs sparingly.
Every step shouldn’t be a screenshot. If your UI is busy or you want testers to find something buried deep, strategically placed screenshots help. It makes it easier for testers to understand how to execute complicated actions OR show them what elements of the page they should be focused on.
Test writing and account seeding
- Keep instructions short, and follow the test writing helpers in-app.
The helpers are there to guide you so you don’t use quotes when not necessary (they can increase verification time if you need testers to look for an exact match), and they encourage you to keep your instructions short.
- Stay away from using internal company jargon.
Stick to language that describes the functionality as it appears to end-users.
- Use of Click to Copy.
Click-to-copy works on text and links and can be added from the app or the CLI. Testers will only have to click to copy the words, phrases or links and paste them into the terminal, which speeds things up and ensures accuracy.
- Have enough reusable step variables.
If you use reusable variables, make sure you have enough of them, or disable that option so you don’t find yourself unable to start the run on time.
- Make sure necessary states are seeded when testers start working.
Whatever this means for your app, it’s much faster.
Example: To test the checkout process. If you have the test seeded so that an account has items in its shopping cart and the tester only has to go through the checkout process instead of the "shopping" and checkout it will go faster. If you invest in proper seeded states upfront, you’ll see a payoff later on when our crowd of testers executes your steps because there’s less for them to do.
We hope this provides guidance to help speed up runs. If you have any questions, please email us at firstname.lastname@example.org.