How Rainforest Algorithms evaluate test results
The Rainforest algorithm is constantly evaluating and learning from collected behavior and results. Rainforest recruits at least 3 testers for every test. Each tester’s actions—such as mouse activity, time spent on each step, and correct site navigation—are all monitored by our system and compared to one another to ensure consistency.
How are Acceptance and Rejections determined?
Based on our algorithm and overall evaluation of a result, Rainforest will either accept or reject the test results of each tester. Accepted and rejected results are not the same as the "Pass" or "Fail" results for test steps. Instead, this is the means by which we determine our confidence in the accuracy of the submitted test result.
- Results determined to be correct are accepted and are taken into account when reporting the final state of the test (Pass/Fail) to our customers.
- Results that cannot be passed with confidence are rejected. These results are not taken into account in the final (Pass/Fail) determination of the test itself, but the result and reason for rejection are still displayed for you to evaluate.
Why do some tests have more than 3 assigned testers?
If a tester result is rejected, we may recruit additional testers until we have a consensus on whether the test has passed or failed. Although it is not typical, tests can sometimes recruit up to 12 testers.
Our algorithm: Simple Vote Multi-Fail (SVMF)
- Our current algorithm looks for 2 matching results (consensus) or seeks out 3 conflicting failures.
- 2 matching results, either 2 passes or 2 failures, is interpreted as a "consensus" result. When two testers successfully execute a step in a browser and answer with a 'yes', this step is deemed as having "passed". If two testers fail the same step by answering 'no', that step and the test as a whole in that browser will be judged as having "not passed/failed".
- For 3 conflicting failures, if 3 testers fail a test at 3 different steps, then the test result is deemed to not have passed. If this were Simple Vote, the system would wait until 2 matching results came up, which typically translates into longer run times.
- The test results page will show multiple "X"s on a test that failed at multiple steps. This will indicate where testers failed a test which leads to the overall failure.
- Since there is less need to recruit more testers in a given test, tabular variables will be less likely to be exhausted during a run, preventing unnecessary run errors.
Why are some results rejected?
This is a common question, and you may notice some tester that completed the test had their results rejected. Results may be rejected for the following reasons:
- The tester’s results do not agree with the other testers.
- The tester completed the test too quickly for us to be confident in the result.
- The tester did not navigate to the URL defined in the test steps.
- A Rainforest admin rejected the tester’s results after reviewing their work.
- The tester failed to perform the expected actions (clicks, scrolls, mouse movements, etc) over several steps.
- The tester failed a built-in quality control test.
- For certain accounts that have the algorithm set as 'Paranoid' -- if a single tester has reported a failure, all passing results will automatically be rejected. Paranoid mode is less commonly used, please reach out to email@example.com if you have questions about the algorithm that is set for your account!
Why do I see different numbers of testers in my results?
Rainforest tests will always use a minimum of three testers per platform to produce a result. In some instances, more than three testers are required to qualify a result. Some reasons for pulling in extra testers include:
- Rainforest detects ambiguity in test results
- Rainforest detects disagreement between 3 original testers for a platform
- Rainforest detects that a tester has exited their task prematurely
- Rainforest detects that a tester has been idle for longer than 15 minutes
- Rainforest detects that a run is taking longer than expected to complete
- Rainforest detects significant discrepancy in the meta-data (time on step, button clicks relative to other testers, etc...) between testers sent to a run
If you have any questions regarding the results of a particular test, please reach out to firstname.lastname@example.org!