All results can be found from your Results page. Let's break down what you will see.
The leftmost column of the results dashboard begins with a set of colored circles that allows you to quickly identify the status of the run.
Green = The run is complete and passed
Red = The run is complete and failed
Yellow = The run was aborted
White = An error caused the run to not start
On the homepage of results, there are various columns of information. The information is presented is below moving from the left column to the far right column.
Creating a run description is optional when running your tests to keep yourself and team on the same page on details of the specific run (For example, Round 2 or Pre-release).
Browsers or Run Groups
In the 4th column, select the column title 'Run Groups' and you can toggle the information presented to either the browsers that were included in that run or the name of the run (Run Group or Feature name).
Start Time, Duration, and SpeedUp
In the 'Start Time' column, you can toggle between the start time of the run, the duration of the run, and the speedup metric for that run.
Start time indicates when the run was triggered dynamically or triggered by a schedule.
Duration is how long it took for all the results in the run to come back with a consensus on Passed or Failed.
Speedup is the amount of time total it took our testers to complete the tests (if they had to be run sequentially) versus the amount of time you had to wait for the run in Rainforest since they were run in parallel. This allows you to understand how much faster Rainforest may be versus having an in-house testing team run your tests cross-browser.
This will break down all the tests in the run by passed, failed, or other. The number of passed tests is in green, the number of failed tests is highlight by a red box. The reason for an “other” or the greyed out result would be if a run had an error or was aborted.
Breaking down Run Results
When you click on a run you will land on a page that may look like this:
Automatically, the failed results are surfaced to the top of the list and the platforms the test failed on indicated by platforms icons in red on the right side of the page.
Next to the icons is a column that displays the test step(s) that failed during that run. The number shown indicates exactly which step number failed and if you hover over the step number a modal will appear displaying the exact step instructions and the platforms it failed on.
If you proceed to click on any of the red platform icons or the view results link you will be brought directly to the step that failed to see the actual results.
There is a left arrow icon next to the search box on the right-hand side the overall information about the run is presented. Select that arrow and the Run Log / Run Info is displayed.
The log updates events during the progress of a run. It is a helpful section to see what is occurring during a run while you wait for it to complete.
Rerun All Failed Tests from Individual Run Level
This button allows for the option to rerun all the failed test in the given run once more.
Breaking down Detailed Test Results
The center of the page will show you the Test Title.
- Click on “Go To Test” to go back to the test view and edit step
- Click on “<” button to go back to the run results list view. Re
Rerun An Individual from the Test Level
Click on the drop-down arrow next to Rerun and you can choose from the following options after you viewed your test failures:
- Rerun all browsers (regardless of passed / failed)
- Rerun all failed platforms
- Rerun *specific browser*
Individual Test Results Breakdown
Click into one of the 'X' icons, from this view, you can drill into any step for each tester and take additional actions.
Navigate between steps
You easily use your keyboard to cycle between all steps to see tester results.
- Click on any step, which will bring up the screenshots of all testers and their results
- You can click on each screenshot to get a more detailed view of the screenshot, which includes browser information, the tester ID and comments, along with links to download test logs.
- Use your keyboard’s Left/Right arrow keys to cycle between testers screenshots. The UP/Down arrow keys will cycle screenshots between steps. This is a great way to track the actions any individual tester.
- Do you see three dots on any steps (...)? When you click on them, you will see any screenshots of testers that had their results rejected.
You can manually change the result of any step to impact the overall results of the test.
- Click on a step, you will see a button labeled “Change to passed” or “Change to failed.”
- When you click on the box you will be prompted to give a brief description of why the change is needed.
- This description is saved and displayed along with a timestamp which will help communicate changes to your team.
- The change will impact the result of the browser test.
Breaking down by Screenshot
Screenshots, Mouse Tracking and Logs
- The green eye icon toggles mouse tracking on and off.
- Once you click into a screenshot, you can see tester comments.
- On the bottom right corner, you can download the HTTP Archive logs (HAR).
How to incorporate reviewing results in your workflow
- Integrate with Slack to view failures in a Rainforest channel and get immediate feedback on webhook timeout, run failure, or run passed.
- Integrate with JIRA/Pivotal to view failures as a ticket in a new Rainforest project - more readily move a bug into the product and development team’s workflow to take next steps
- (Coming Soon) Click on the “View <username> Log File” from each screenshot to see the exact network requests sent to your application to identify which request failed even faster.
- If you use PagerDuty, implement an alert that will let your team know if there are unreleased issues to review and fix. Check out how Rainforest implements this for our development team here: https://github.com/rainforestapp/release-police