FAQ

How many automation agents are allocated?

We allocate up to 2 agents for each test. Though we only surface one result, a second agent might run the test when we update our Automation Service. The test executes on the previous and new versions of the automation. If the test passes, the test upgrades to the latest version of our Automation Service.

How many automated tests can run in parallel?

As many as your testing environment can handle. There’s no limit to the number of automated tests (or tester community tests) that can run in parallel.

How does Rainforest’s Automation Service perform its evaluation?

Rainforest Test Automation is pixel based, which means we test what your user sees and how they interact with your site. On a practical level, some things are impossible to accomplish with DOM-based tools, which test what the computer sees, not the user. These include:

• Working across browser windows
• Interacting with the OS
• Downloading, editing, and uploading files
• Any interaction with desktop software

When looking at my tests, which tests are a good fit for the Automation Service? Why would I want to use manual execution vs. automation?

A good rule of thumb is to use Plain-Text Editor tests for parts of your product that are not yet stable. Suppose you’re in the middle of introducing a new design language or just starting to build out a given area. In that case, human testers are better at understanding ambiguities. They can also spot problems, such as a missing image or an alert box at the top of the page the Automation Service might not catch. Moreover, human testers can leave comments, providing additional information.

Similarly, if your application or testing environment is dynamic or unpredictable (the latest news, streams of recent images, relative dates, pop-up windows appearing at random intervals), you have more success with human testers. You can still write your tests using the Visual Editor interface, though, giving testers clear instructions.

For mature parts of your product, where you know well what you want to test and your environment is predictable, automation is a better fit. It’s faster, and you can run it much more frequently, which can speed up your release cadence.

What are the limitations?

• Not compatible with mobile.
• Optimized for a single browser, meaning that the test runs with the browser it was created for when executed by automation. When performed by humans, it’s compatible with multiple browsers.
• Dynamic data can be used for data entry via placeholders but can’t check subsequent screens.
  For example, if the dynamic user is “Fred,” there’s no way to validate that Fred is logged in.
• Calendar and other mathematical manipulation.
  For example, you can’t advance the date by 3 days or confirm the correct sales tax was added.
• There’s no DevX/RFML equivalent for Visual Editor tests.
  However, tests can be executed via the CLI. For more information, see Running Tests from the CLI.
• Automation can’t execute steps requiring human judgment (“Is this a picture of a dog?”).
  However, you can add those steps into your Visual Editor tests using the Tester Community action. Any test with Tester Community actions can only be executed using our tester crowd. In the future, we plan to support a hand-off between automation and humans within the same test.

If you have any questions, reach out to us at [email protected].