How to Categorize Failures

Learn how to use failure categories to provide more meaningful data to your team.

Failure categorization is essential to making sure you get the most information and understanding of your test suite’s health. It also ensures the proper action is taken when a test fails. Failure categorization is also used to populate the Weekly QA Visibility Report and report issues to the Rainforest Support team.

Failure Categories

Following are the six failure categories you can use.

Category

Description

Bug

An issue that needs to be addressed. The feature or function doesn’t work as intended.

Known Bug

A bug that was previously identified.
Note: Use Bug to categorize the first instance of the issue and use Known Bug for any additional tests or run that resurface the same known bug.

Needs Refactor

The test needs to be updated (rewritten) to match the tested functionality.
Note: If the Test Writing Service is part of your suite, you can rewrite the test yourself or ask Rainforest to rewrite it for you. For more information, see Using the Test Writing Service.

Tester Error

The system doesn’t have a bug. Though the instructions were correct, the tester failed the test or didn’t follow the instructions.

System Issues

There was an issue with the system, such as connectivity or incorrect data. You can select whether the problem is in your environment or in Rainforest’s VM.
Note: Selecting the “Rainforest VM or device” option reports the issue to our Support team.

Other

A catchall category for other items that might have caused the failure.

Accessing Failure Categorization

Failures are highlighted on several pages. You categorize failures either on the Run Summary page or the individual Test Results page.

The Dashboard Page

The Dashboard page shows issues categorized as Bug or Needs Refactor over the last seven days. Selecting a test from the list takes you to a page where you can review and update.
Clicking Go Categorize takes you to the Run Results page, filtered to show failed tests. From there, you can categorize your tests.

Categorizing a failure.Categorizing a failure.

Categorizing a failure.

The Run Results Page

Selecting a test run from the Run Results page takes you to the Run Results Summary page, where you can categorize the failures.

Accessing the Run Results page.Accessing the Run Results page.

Accessing the Run Results page.

The Run Results Summary Page

The following steps show you how to categorize failures on the Run Results Summary page.

  1. Click the checkbox next to the tests you want to categorize.
    Failure categorization is only available for tests that have failed.
  2. Click the Categorize Failure dropdown.
  3. Choose the appropriate failure category.
    Note: When you click multiple checkboxes simultaneously, the failure category, comments, and assignments are applied to all selected tests.
Categorizing a failure via the summary page.Categorizing a failure via the summary page.

Categorizing a failure via the summary page.

The Automation Services Test Results Page

After a test executes via the Automation Service, you can categorize any failures from the Test Results page.

The Automation Services Test Results page.The Automation Services Test Results page.

The Automation Services Test Results page.

The Tester Community Test Results Page

After a test executes via the Tester Community, you can categorize any failures from the Test Results page.

The Tester Community Test Results page.The Tester Community Test Results page.

The Tester Community Test Results page.

When selecting the failure type, you can provide additional information in the Failure Category field (up to 250 characters). Team members can view the information by hovering over the categorized issue. You can also alert a team member by using the Notify Team Member dropdown. Doing so sends an email about the issue.

To remove a failure category from a test, click the Failure Categorization dropdown, then select “Clear Category.”


If you have any questions, reach out to us at [email protected] or via the chat bubble below.


Did this page help you?