Yes. When running Visual Editor and Plain-Text Editor tests as part of the same group (Run Group, Feature, or Saved Filter), you have the option of “Crowd Only” or “Crowd + Automation.” The automation bot executes the Visual Editor tests, and the Tester Community runs the Plain-Text Editor tests.
We assign a minimum of 2 testers per browser for each test. Sometimes, our algorithm determines that additional testers are required to reach a consensus, or testers abandon a test in progress. In these cases, we automatically recruit more testers to ensure a reliable result.
We recommend creating a sufficient number of accounts to accommodate everyone assigned to your test. Here’s an easy way to determine the number.
(Tests x Browsers x 2) x 3
- Tests is the number of tests.
- Browsers is the number of browsers.
- 2 is the default number of testers for each test.
- 3 is a safety margin in case we need to add more testers.
Let’s say you want to run 1 profile update test using 5 browsers. That requires a minimum of 30 test accounts. (1 x 5 x 2) x 3 = 30.
To ensure they can access your environment, Rainforest testers come from various IP addresses. For more information, see Which IP Addresses Do Rainforest Tests Come From?
Yes. Use the Geolocation option to specify countries such as the United States and Canada. For more information, see Executing Tests Using Specific Geolocations.
If it’s already enabled for your account, the Location dropdown appears under Additional Options when you run your tests. To request this feature, contact [email protected].
Yes. Multiple testers can appear to come from the same IP address, especially when testing using the global crowd. Note that the Geolocation option restricts testers to a single IP address.
A good rule of thumb is to use Plain-Text Editor tests for the parts of your product that are not stable. For example, let’s say you’re in the middle of introducing a new design language, or you’re just starting to build out a given area. In these cases, a human tester is more forgiving. They understand ambiguities.
Humans can spot problems you weren’t expecting, such as a missing image or an alert box located at the top of the page that a user might not notice. Human testers can leave comments on your tests, providing more information and pointing out potential problems.
Similarly, human testing is more suited to applications or environments that are dynamic or unpredictable. Examples include the latest news, streams of recent images, relative dates, pop-up windows appearing at random intervals. Note that you can still write your tests using Visual Editor to provide clear instructions for testers.
For the mature parts of your product, where you know what you want to test and where your environment is predictable, automation is a better fit. It’s faster, and you can run it more frequently, which can accelerate your release cadence.
If you have any questions, reach out to us at [email protected].
Updated 4 months ago