How our Algorithms determine how many testers are needed

With our Simple Vote Multi-fail algorithm, a minimum of 3 testers are assigned to each test per browser.

For example, running 1 test on 1 browser will send at least 3 testers. Running 1 test against 3 browsers will send 9 testers. Running 3 tests against 3 browsers will have a minimum of 27 testers, and so on.

If our algorithm determines that additional testers are needed to reach a consensus, or testers abandoned a test in progress for whatever reason, it will recruit more testers, up to a total of 6 testers per browser the test is run against, to deliver a reliable result for the test.

If you are setting up variables for your tests, a good rule of thumb is to count Number of tests containing variable * number of browsers testing across * 6 (testers).

Why?

This is the most basic method of Quality Control we do — if all testers reach a consensus on the outcome of the test, then we return the result to you.

To learn more, check out how we algorithmically evaluate results!

Did this answer your question?