This article is intended for folks with relatively undefined Quality Assurance (QA) processes. If you’re looking for actionable guidance on developing an intelligent, scalable approach to QA, this guide is for you. In it, we explore:
- Strategy. How to think about QA and where Rainforest fits.
- Coverage. How to plan and build your initial test suites.
- Execution. How often to run your tests using different browsers.
- Results. How to triage test failures, communicate defects to your team, and measure your efforts.
If you’re new to QA, there are lots of online resources that explain the basics. Here are a couple to get you started:
QA is a function dedicated to risk mitigation. The goal is to deliver a quality product to your customers. To do this, QA must proactively and continually assure quality at every stage of the development cycle. The later defects are caught, the longer and more expensive they are to fix. So, the process must be designed to catch defects as early as possible.
A fundamental principle is prioritization. You have limited resources (time, money, and people). Therefore, it’s critical to invest in testing activities that yield maximum returns while avoiding activities that don’t provide equivalent value. In other words, there’s a line of diminishing returns when it comes to QA testing. The following diagram does an excellent job of illustrating the point. If any terms are unfamiliar to you, read on.
Rainforest does functional testing at the UI level. In other words, we validate your software using functional requirements and specifications.
In the typical evolution of a QA function, testing begins as an ad hoc, manual activity. Often, there are no documented test cases. As QA matures, testing becomes more structured, and the team develops a process. Over time, the product grows in size, and QA can no longer keep up with development.
Rainforest helps customers at every point in their QA evolution to achieve speed, efficiency, and scale. We do this by offloading manual and repetitive testing. For more information about our core product offering, visit our website.
Don’t have any test cases? Or, if you do, you’re not confident of their effectiveness? If you said yes to either of those questions, you need to begin with What should I test and how should I test it?
When starting out, most people make the mistake of testing everything. However, like many things in life, testing follows the Pareto principle, better known as the “80/20 rule.” In QA, a small number of tests provides the majority of the value. Therefore, it’s essential to prioritize when building your initial test suite.
Throughout this exercise, refer the utility pyramid below. The size of each segment represents the value the tests provide, not the total number of tests.
We suggest a feature-driven testing approach focused on building coverage that maps critical use cases and key features. Here’s how to do it:
- Build Your Smoke Suite
Begin by considering your entire application as a single feature. Next, identify your top 5–10 most critical user scenarios, such as add to cart, make a purchase, and sign up. These tests are your smoke suite. You run them early and often to validate your app’s quality.
- Build Your Regression Suite
Regression suite is a generic term, but it includes all tests across the entire utility pyramid. You run this suite before releases to ensure changes to your app don’t introduce regressions. When building it, start small, and remember to prioritize.
Start by breaking down your app into features. Identify the most common user scenarios within each feature. It can be helpful to map your app in a spreadsheet. Check out this example.
When writing a test, it’s tempting to include everything. Ideally, tests should be short, modular, and confined to a specific function or user flow. For more information, see Ideal Scope of a Rainforest Test.
More tests don’t always equate to more coverage. In fact, we’ve found that utilizing a smaller group of tests strategically is far more impactful than building a huge database. To learn how to use Rainforest’s test organization features, see Deep Dive: Building Your First Test Suite.
Now that you’ve written your initial test suite, it’s time to determine how and when to run the tests. Your smoke suite is a small set of your most critical tests that you run early and often to determine if a build is stable enough to proceed with further testing. Run your smoke suite at least daily as part of an automated schedule. Or integrate them into your CI pipeline and run on every merge to your release branch.
As a best practice, we suggest running your smoke suite with only your most popular browser/platform to get quick, inexpensive feedback.
Your regression suite is comprised of all of your tests. Before every major feature release, run tests as far down the utility pyramid as possible (smoke + happy path + regression tests) for all features. If [integrated into your CI pipeline, run your smoke tests along with any other tests that touch the updated features.
As a best practice, we suggest running with the latest version of the “Big Four” (Safari, Microsoft Edge, Chrome, and Firefox. Then, expand based on your user traffic sources.
For engineers, run your test suite from the command line using our Command Line Interface (CLI). Most customers choose a combination of our CLI client and a cloud-hosted CI provider. For example, we use CircleCI, but there are many others.
It’s common to run all builds that are triggered by a release. In Rainforest’s case, that’s the Develop > Master branch merge. However, you could run all builds, all builds with a specific tag, or all builds triggered by a particular branch.If you’d like help with a custom requirement, get in touch with our Support team.
Testing in Production may be the easiest choice, but it’s too late to be effective. Your customers might have found bugs already. However, testing in Production does provide assurance that things are working for your users. To find issues before your customers do, you must have a highly streamlined testing process in place.
We recommend testing in a dedicated QA environment that mirrors Production as closely as possible. Testing before release acts as a gate check to catch defects before they reach your users. Moreover, doing so provides you with early feedback.
As you mature your testing process, you might consider using seed dataseed data - Information you provide that’s required for testing your application. Examples include usernames, passwords, and roles. to allow for more complex state testing with less effort. Webhooks are another option. For more information, see these articles:
The value a test provides is found in the results. Ensuring you have an effective process for investigating and triaging failures is crucial When analyzing a test failure, it’s imperative that ownership is clear. You must identify the person responsible, as well as the scope of that responsibility. In this section, we cover:
- Receiving triage results
- Documenting bugs
- Communicating bugs back to the team
- Measuring quality over time
Pushing test results notifications into the channels your team already uses ensures quality issues remain front and center. Suppose the team uses chat apps such as Slack or Microsoft Teams. In that case, we suggest setting up results notifications integration.
Sending notifications to team channels is a great way to keep the team engaged with QA and ensures no issues fall through the cracks. For more information, see Integration Partners.
Note: If you don’t use a chat app, you can still set up email notifications in your Rainforest settings.
Every failure should be investigated and categorized. Moreover, you should take action on the failure category immediately. This includes:
- Updating an outdated test and rerunning it
- Fixing data within your own test environments
- Reporting a bug
Deferring test maintenance creates noisy test results. Often, this results in the team losing trust in the test suite, or even worse, missing bugs. To ensure your test suite is clean, stable, and up to date, we recommend enforcing a policy of categorizing 100% of test failures. Doing so ensures all failures are investigated. Moreover, it creates an internal SLA for how quickly the team is expected to recover from a failure.
After diagnosing the cause of a defect, feedback must flow back into the appropriate team. First, ensure you have a bug classification standard that’s understood and agreed upon by QA and developers. Next, begin tracking all defects in a shared spreadsheet or an existing bug-tracking tool. Defects are recorded, and a defect’s lifecycle can be referenced anytime by team members.
If you use Jira, consider setting up Jira integration so your team can quickly and easily export failures, including logs, screenshots, and links back to the test failure.
The primary goal of a QA process is to mitigate risk when shipping new code. Therefore, you must measure your results to learn about the current state of quality. Moreover, you must measure improvement. Primary measurements include:
- Number of Bugs. Log any reported issues by date, product area, priority, and issue source. For example, a source could be external (customer), internal (missed by QA), automatic (error reporting), or a test-case failure. Regularly summarize this log, look for patterns, and report back to the team.
- Time to Fix. This measurement provides an answer to how a development team can use the output from QA to triage and fix bugs. The simplest way to measure is by determining the interval between a failed build and the next passing build.
- Time to Test. Which tasks and activities take you the longest? Do they provide equivalent value? What can you offload? Rainforest helps you scale your testing efforts, measure and track your progress over time, and discover patterns.
We know that building a strategy from scratch can feel like a daunting task, and we hope this guide serves as a roadmap to help you figure out where and how to focus your efforts.
If there’s one thing you should take away from this guide, it’s this: QA is an exercise in risk mitigation. Only you know the level of risk tolerance your organization has when it comes to quality. To summarize:
- Know what your quality standards are.
- Start small.
- Focus on investing in the testing activities that provide maximum value.
- Measure and track your progress along the way.
If you have any questions, reach out to us at [email protected].
Updated 4 months ago
- Ideal Scope of a Rainforest Test
- Deep Dive: Building Your First Test Suite
- Running Tests Using Schedules and Run Groups
- Using the CLI for Continuous Integration
- Seeded State Management
- Using Webhooks
- Integration Partners
- How to Categorize Failures
- STF: Software Quality
- The Top 5 Don’ts of Software Testing