Failure categorization is easy and essential to ensure that you get the most information and understanding of the health of your test suite.  It also ensures that proper action is being taken when a test does fail.  Failure categorization is also used to populate the weekly QA Visibility Report and to directly report VM issues to our Support Team.   

Failure Categories

The following are the five failure categories available.

  • Bug - An issue that needs to be fixed in the environment being tested
  • Needs Refactor - The test needs to be updated (rewritten) to match the functionality being tested.
    If Test Writing Service is part of your suite, you may select to rewrite the test yourself or have Rainforest rewrite it for you. Click this link to learn more about Test Writing Service!
  • Tester Error - The system doesn't have a bug, the instructions were correct, however, the tester incorrectly failed the test.
  • System Issues - There was an issue with the system such as login credentials, connectivity, or incorrect data. You'll have the option to select if the issue is with your own environment or a VM issue on Rainforest's side.
    NOTE: Selecting the "Rainforest VM or device" option will report the issue to our team for investigation.
  • Other - A catchall for any other items that could have caused the failure

Navigate to the Results tab on the left panel then click into a past run to be advanced to the Runs Summary Page, where you can actively categorize the failure
At the Runs Summary page and the Test Results page you can categorize failures directly.  

Accessing Failure Categorization 

Failure Categorization is highlighted on several pages

  • Dashboard Page
  • Runs Page
  • Runs Summary Page
  • Test Results Page  

Dashboard Page

Selection of Go Categorize > advances you the Runs Page with a filter that shows failed tests/runs.  
The selection of any line will advance you to the Run Summary Page where you can actively categorize any failures.

Runs Page

The selection of any test result will advance you to the Run Summary Page where you can actively categorize any failures.

Runs Summary Page

Actively categorize failures by selecting the Needs Categorization option. 

Test Results Page

Actively categorize failures by selecting the Categorize Failure dropdown

Failure Categorization Categories and actions

Categorization on the Runs Summary page and the Test Results page work similarly.

  1. Select Needs Categorization 
  2. Select the proper category

Upon selecting the issue type, you may provide additional information in the Failure Category dialog box (up to 250 characters).  This information will be visible to any teammate by hovering over the categorized issue.  You may also Notify a team member from the dropdown selected.  This teammate will receive an email about the issue.

NOTE: if an issue is re-selected and SAVED, ALL prior Failure Categorization dialog box information will now not be visible. 

If you categorize a result as “needs refactor”, you have the option to rewrite it yourself or send in the test for our trained test writers to update. If you choose to rewrite it yourself, you can also set it to draft state. Any test set to draft state will show up in the “Draft Tests” view, and it won't be run until the test is published.

If you choose to send it to our team of test writers to be updated, leave a detailed note about what needs to be updated. Visit this link to learn more about our test rewriting feature!

As issues are categorized for a run, the toucan will fill with more color and the Failure Category Breakdown section will update how the failures were categorized. Just another way Rainforest makes QA enjoyable.

Hovering over the issue displays the detailed information and the team member that was notified (if provided).  

Did this answer your question?