Balancing Automation with Manual Testing

Overview

Okay, you’ve read our post on QA Strategy, and now you’re ready to build and run your smoke suite, happy path, and regression tests.

We support two different test types—those written using the Visual Editor and those created with the Plain-Text Editor. You don’t need to choose just one; Rainforest allows you to combine both. Use both for a balanced testing strategy that takes advantage of human judgment and flexibility where needed and the speed/cost savings of automation when it’s not.

In this article, we cover:

• The tradeoffs you should consider as you evaluate which tests you plan to write using Visual Editor and when to execute them using Rainforest Automation or the Tester Community
• Which types of tests you can’t run using automation, as well as which tests can be run with automation but probably shouldn’t be
• Why you might want to write your tests using either Visual Editor or Plain-Text Editor
• Which types of tests are well suited for Rainforest Automation and Tester Community testing

Factors to Consider When Writing and Running Your Tests

Smoke/Sanity Testing

Smoke tests are the core business-critical tests you run frequently—maybe several times a day if you hook them into your CI tool. Because these tests constitute your core functionality that likely doesn’t change often, you should automate them whenever possible. Thus, you can run them regularly without driving up costs—especially if you hook the tests into a CI pipeline. Moreover, you benefit from faster run times than you could expect from human testers (4 minutes versus 17 minutes on average).

In summary, write these tests using Visual Editor so they can be executed by Rainforest Automation.

Happy Path and Regression Testing

These tests are designed to ensure product stability. However, because you run them less frequently than your core smoke suite, the benefits of automation are less apparent. Of course, faster and cheaper is usually better, but automation might not be better suited in this case.

As with all scripted testing—especially automated testing—test cases can break as your app changes. However, tests run using the Tester Community are less likely to fail after making small tweaks such as minor copy or layout changes. Of course, keeping tests up-to-date is always a best practice.

Rainforest testers are more flexible when running a Visual Editor or Plain-Text Editor test. So, you can save yourself some maintenance by using the Tester Community for happy path and regression testing.

Mobile Flows

Always test mobile functionality via the Tester Community. Though Rainforest plans to extend automation to mobile web and native app testing, we’re not there yet.

Dynamic Flows

Sometimes tests leverage data that changes between runs, such as when you need to verify a username that's automatically generated and is never the same twice. These workflows cannot be tested via automation because they’re not deterministic; you can’t validate a screenshot of a UI element when the data or text consistently changes.

Flows That Require Human Judgement

Some items require a person to exercise their own judgment. This is why having a balanced approach is better. Examples of items not suited for automation include CAPTCHA forms and test cases that require performing calculations or comparisons.

Compatibility Testing

Rainforest tests written with Visual Editor rely on screenshots taken on a specific platform. Running these tests on multiple platforms using Rainforest Automation frequently exposes minor changes in how your website or application renders that don’t actually represent functional bugs. For example, your UX designer decided to change the text of a button from “Submit” to “OK.” Functionally, clicking the button triggers the same action, but the test fails.

Tester Community Testing—Which Editor to Use?

When getting started with Rainforest, we recommend giving both editors a try. Tests written with the Visual Editor can be run using both automation and the Tester Community. When choosing the latter, we translate the actions and assertions into steps that can be performed by our testers. If there are tests that you plan to run using both methods of execution, writing them with Visual Editor allows you to leverage both.

With tests not suited for automation, you have a choice of editor. Use whichever one you prefer. The Plain-Text Editor requires writing your tests using free-form English, which allows the Tester Community to execute them quickly and efficiently. Visual Editor is intuitive and includes a Rainforest Virtual Machine to help guide you in writing your test cases, even if you don’t plan to automate them.

Summary

Rainforest gives you the tools to create a testing strategy that combines the best of both automated and manual, crowdsourced testing. We recommend that you:

• Automate your smoke suite and run it on one platform once daily or multiple times per day.

• Utilize the Tester Community for use cases not suited for automation.

• Determine how much value automation provides when weighed against the cost of maintenance for your non-business-critical tests. The less frequently you run these tests, the lower the ROI for maintaining your automated tests. Take advantage of the flexibility of the Tester Community to save time on maintenance.

• Try out both methods of test writing for tests that are run exclusively using the Tester Community.

If you have any questions, reach out to us at [email protected].