FAQ

Can I run a group of tests using the Tester Community and Automation at the same time?

Yes. When running Visual Editor and Plain-Text Editor tests as part of the same group (Run Group, Feature, or Saved Filter), you have the option of “Crowd Only” or “Crowd + Automation.” The automation bot executes the Visual Editor tests, and the Tester Community runs the Plain-Text Editor tests.

How many testers run my tests?

We assign a minimum of 2 testers per browser for each test. Sometimes, our algorithm determines that additional testers are required to reach a consensus, or testers abandon a test in progress. In these cases, we automatically recruit more testers to ensure a reliable result.

How many test accounts do I need?

We recommend creating a sufficient number of accounts to accommodate everyone assigned to your test. Here’s an easy way to determine the number.

👍

(Tests x Browsers x 2) x 3

• Tests is the number of tests.
• Browsers is the number of browsers.
• 2 is the default number of testers for each test.
• 3 is a safety margin in case we need to add more testers.

Let’s say you want to run 1 profile update test using 5 browsers. That requires a minimum of 30 test accounts. (1 x 5 x 2) x 3 = 30.

Which IPs do testers come from?

To ensure they can access your environment, Rainforest testers come from various IP addresses. For more information, see Which IP Addresses Do Rainforest Tests Come From?

Can I limit the IP addresses by country?

Yes. Use the Geolocation option to specify countries such as the United States and Canada. For more information, see Executing Tests Using Specific Geolocations.

How can I enable Geolocation?

If it’s already enabled for your account, the Location dropdown appears under Additional Options when you run your tests. To request this feature, contact [email protected].

Can testers come from the same IP address during the same run?

Yes. Multiple testers can appear to come from the same IP address, especially when testing using the global crowd. Note that the Geolocation option restricts testers to a single IP address.

How else can I identify network traffic coming from the Rainforest VMs?

See Which IP Addresses Do Rainforest Tests Come From?

How can I identify webhook traffic from Rainforest?

See Which IP Addresses Do Rainforest Tests Come From?

Why would I want to use manual execution with Plain-Text Editor vs. Visual Editor tests?

A good rule of thumb is to use Plain-Text Editor tests for the parts of your product that are not stable. For example, let’s say you’re in the middle of introducing a new design language, or you’re just starting to build out a given area. In these cases, a human tester is more forgiving. They understand ambiguities.

Humans can spot problems you weren’t expecting, such as a missing image or an alert box located at the top of the page that a user might not notice. Human testers can leave comments on your tests, providing more information and pointing out potential problems.

Similarly, human testing is more suited to applications or environments that are dynamic or unpredictable. Examples include the latest news, streams of recent images, relative dates, pop-up windows appearing at random intervals. Note that you can still write your tests using Visual Editor to provide clear instructions for testers.

For the mature parts of your product, where you know what you want to test and where your environment is predictable, automation is a better fit. It’s faster, and you can run it more frequently, which can accelerate your release cadence.

If you have any questions, reach out to us at [email protected].