FAQ

Frequently asked questions about running tests using the Automation Service.

Are Visual Editor tests code based?

A Visual Editor (VE) test does not require any code. This design makes it easy for technical and nontechnical teammates alike to own testing. Moreover, it offers flexibility, which Rainforest users value. VE tests work on a visual level. Users interact with the UI and not with the DOM.

Rainforest’s Automation Service runs your application automatically on our scalable virtual machine (VM) infrastructure. Your application is tested visually by comparing the results on the screen with those in your test. And all without code.

How does the Automation Service perform its evaluation?

The Automation Service is pixel based vs. DOM based, which means we test what the user sees and how they interact with your site.

On a practical level, some things are not possible with DOM-based testing. These include:

  • Working across browser windows
  • Interacting with the OS
  • Downloading, editing, and uploading files
  • Any interaction with desktop software

No code-based system does this reliably, but our Automation Service can.

The other difference is more philosophical. Purely visual testing evaluates what a user sees and experiences. DOM-based tools test what the user’s computer sees. In some cases, this distinction is crucial, depending on your application.

Can Visual Editor tests run using humans?

Visual Editor enables Rainforest customers to run the same no-code automation test using our robot automation workers or our human crowd. From a single test suite, you can mix and match execution based on your workflow needs. For example, run with humans for major product releases and run every branch merge with robots.

Can I run tests using the Tester Community and Automation at the same time?

Yes. When running Visual Editor (VE) and Plain-Text Editor (PTE) tests as part of the same group, you have the option of “Crowd-Only” or “Crowd + Automation.” The automation bot runs VE tests only once. But you can run those same tests with our community, where at least two testers execute them.

Which of my tests are a good fit for automation? Why would I want to use manual execution vs. automation?

A good rule of thumb is to use Plain-Text EditorPlain-Text Editor - A tool that allows you to write tests as step-by-step instructions in free-form English. Tests created with the Plain-Text Editor (PTE) run using either the Tester Community or your own on-prem testers. tests for the parts of your product that are not stable. For example, let’s say you’re in the middle of introducing a new design language, or you’re just starting to build out a given area. In these cases, a human tester is more forgiving. They understand ambiguities.

Humans can spot problems you weren’t expecting, such as a missing image or an alert box located at the top of the page that a user might not notice. Human testers can leave comments on your tests, providing more information and pointing out potential problems.

Similarly, human testing is more suited to applications or environments that are dynamic or unpredictable. Examples include the latest news, streams of recent images, relative dates, pop-up windows appearing at random intervals. Note that you can still write your tests using Visual Editor to provide clear instructions for testers.

For the mature parts of your product, where you know what you want to test and where your environment is predictable, automation is a better fit. It’s faster, and you can run it more frequently, which can accelerate your release cadence.

My team isn’t technical and has never used automation before. How easy is it for nontechnical folks to use your automation?

We built the Automation Service from the ground up to be the best no-code automated testing platform available. We’ve had nontechnical users from the start and continue to improve usability. While we want to stay as close to the release process as possible, you don’t need to know how to code to write good tests.

My test application is dynamic. Some states remain from the last time the user visited. Is there any conditional logic I can use?

There are no conditionals in Visual Editor tests when run in automation. Our Tester Community is better suited to execute these.

Is there a hybrid model where pieces of the tests are automated, and then the crowd testers take over after setup? Or automated teardown scripts to clean up?

Currently, this is not possible. For now, you can have both automated and non-automated tests within the same run, but you can’t mix and match parts of the tests. This is something we are looking into as part of our roadmap.

Are Visual Editor tests compatible with mobile?

Not at the moment. You can run mobile tests using our Tester Community.

Is the Automation Service separate from Rainforest?

No, they are the same. Manage your manual and automated tests on a single platform, or run them together and see the results in one place.

How many automated tests can I run in parallel?

As many as your testing environment can handle. There’s no limit to the number of automated tests (or Tester Community tests, for that matter) you can run in parallel.

Are scroll events captured automatically? Or does that require a manual event addition such as clicks and see?

If scrolling is necessary, add an explicit Scroll action; the automation bot does not scroll automatically.

What tooling powers the test runner? Are you running Selenium on the VM or something else?

Unlike Selenium, which locks you into the browser, ours is a proprietary solution. Our technology allows automation to interact with multiple browser windows, desktop software, and the operating system.

For those who might not use this feature, are you confident that automated and crowd testing are well supported?

A significant difference between Rainforest and other automation vendors is our belief that an excellent overall QA strategy combines the power of human and machine capabilities. We plan to continue supporting and improving the way we work with our tester community.

Our long-term vision is to use automation for boring-for-humans testing and the power of human judgment and experience for higher-level tasks. You can continue doing exclusively human testing, and we’ll keep supporting and improving it.

We have sensitive data. Will running automation tests guarantee that no human has access to our IP and data?

If you need custom guarantees about sensitive data, get in touch with us directly. Currently, executing a test with automation does not expose any data to human testers. In general, though, this is not something we can guarantee for every future test.

What if the page takes time to load? Is there a “wait” function, or does the test fail?

The automation bot has some tolerance. For example, if you ask it to click on a button, it waits up to 30 seconds for this button to appear. If the button appears within the window, the bot clicks it and continues; otherwise, it fails the test. Suppose your test requires a longer waiting period. In that case, you can add a Sleep action, which pauses execution for the specified time.

What are the limitations?

  • Not compatible with mobile at the moment.
  • Optimized for a single browser, meaning that the test runs with the browser it was created for when executed by automation. When performed by humans, it’s compatible with multiple browsers.
  • Dynamic data can be used for data entry via test datatest data - Placeholders that allow you to inject other values into your tests. Rainforest supports three types: dynamic, static, and built-in. but can’t check on subsequent screens.
    Example: The dynamic user is “Fred.” There is no way to check to see that “Fred” is logged in.
  • Calendar and other mathematical manipulation.
    Example: Can’t advance the date by 3 days.
    Example: Can’t confirm that the correct sales tax was added to a random item.
  • There’s no DevX/RFML equivalent for RFA at the moment. However, you can execute tests via CLI. For more information, see Running Tests from the CLI.
  • Automation can’t execute steps that require human judgment (“Is this a picture of a dog?”). However, you can add those steps into your Plain-Text EditorPlain-Text Editor - A tool that allows you to write tests as step-by-step instructions in free-form English. Tests created with the Plain-Text Editor (PTE) run using either the Tester Community or your own on-prem testers. (PTE) tests. PTE tests can only be executed by our Tester Community. In the future, we plan to support a hand-off between automation and humans within the same test.

What are the known image matching limitations?

Visual Editor tests employ image matching at the UI level. We do this using grayscale instead of color values, which makes imaging matching more resilient. Color matching can’t be reliably tested today. If you require specific color matching for your test, you should not run Visual Editor tests using automation.


If you have any questions, reach out to us at [email protected] or via the chat bubble below.


Related Articles & Links
Did this page help you?