Testing with the Tester Community

Overview

Rainforest is committed to providing secure, reliable crowd-sourced testing for our customers. Our network of testers is comprised of thousands of individuals from around the world. The Rainforest Crowd is a diverse, well-trained group of people.

Who Our Testers Are

Our testers are located worldwide, with a majority in South America, Europe, and India. They come from a variety of backgrounds, including (but not limited to) teaching, writing, computer science, stay-at-home parenting, and of course, quality assurance.

How We Find Our Testers

Rainforest testers were originally sourced through Amazon Mechanical Turk and CrowdFlower (now Appen). Both services are dedicated to crowd provisioning and management. They provided us with the first check of workers’ identities and payment information verification.

Testers are now a Rainforest proprietary crowd, managed by a dedicated community team. The majority of our tester crowd has been with Rainforest for 5 years. We have not recruited additional testers in recent years. Instead, we’ve focused on improving our existing community’s skills and provide them with enhanced infrastructure and user experience.

The Types of Testers Who Work for Us

In addition to the folks who perform the majority of the tests you see in Rainforest, we also have:

• Exploratory Testers. Highly skilled testers recruited to perform freeform exploratory testing.
• Super Testers. A highly specialized group of testers who help us with various tasks, such as important test quality initiatives and beta testing of new tester-facing features.
• Test Writers. A subset of super testers (also known as test authors) who write tests for clients. These individuals have been trained on the various aspects and nuances of Rainforest test writing.

Managing Testers

Our tester community is managed by a dedicated community team that works closely with product managers and engineers to create features that improve our testers’ everyday workflow.

Rainforest’s Quality Control

Delivering high-quality results is our number one priority. This is the core of Rainforest, and the following is a high-level guide to how we do quality control (QC).

Reputation Score

Every tester at Rainforest has a reputation score that is constantly adjusted and updated based on their work. This is the foundation of our QC process. Factors that impact a tester’s reputation score include:

• Their agreement or disagreement with high ranking testers
• Their agreement or disagreement with low ranking testers
• Performance in our training tasks
• Performance in challenge questions about our rules
• Their response time to new Rainforest jobs

Training

Every tester goes through automated training, both at the beginning of their testing career and on an ongoing basis. This training is multifaceted. Some of the training topics include:

• English language comprehension and writing
• Ability to follow nuanced instructions
• Ability to correctly identify and categorize bugs
• Ability to intuitively understand web interfaces

In addition, the key areas of focus are:

• EXACT adherence to step instructions.
  Testers will only perform the instructions as they are written and have been trained not to take any action that they are not specifically instructed to do so in the current step. This allows you to be absolutely certain that your results reflect exactly the flow you wish to validate.
• EXACT matching of page content to the expectation set in the step
  Testers will fail a step where any number/letter/color/content placement/etc. on the page differs from the question asked. This allows you to ensure that the key components/processes are functioning perfectly correctly.
• EXACT matching of any text content in quotes.
  Testers will fail any step where quoted text differs from the step instructions. Testers will not require text to match exactly if it is not surrounded by quotes. This allows you to direct your tester's focus to copy that must be matched exactly.

If any observation in the process of testing differs from these testing rules, or if the testers are unable to complete the instructions, either because the instruction is unclear or some issue on the page prevents them from proceeding, the testers will end the test and provide clear, actionable feedback on why they ended the test.

Results

We algorithmically review all results from testers in real time. This ensures high-quality results while maintaining the speed we’re known for. We continue to update our algorithm and take the following into account:

• Agreement with other testers (weighted by reputation)
• Past history of test
• Interactions by the tester with your website
• Interactions by the tester across the rest of their system during the test

Peer Review

In order to assure the testers are doing what is requested of them, we have implemented a peer-review mechanism, where our pool of proven and trusted testers review and flag tester work for review by our staff.

These trusted testers do not have the ability to hand out rejections or block accounts -- rather, they help Rainforest admins review test results and provide feedback as to whether the work was accurate or not. Every piece of feedback they give is then reviewed by a Rainforest admin before the decision to reject or block a worker is made. This not only works as quality control, but also as a filter to decide which testers shouldn’t be allowed back into the community.

Optional Rainforest Tester Security Measures

We also took an extra step for our customers' comfort and have implemented a series of optional requirements specific to said customer account. Only testers that have meet the requirement or requirements requested would be able to access customer tests.

Malware Scans

Our testers can provide recurrent malware scans from the systems they use to access our platform. We support Windows through Windows Defender and Mac/Linux systems through ClamAV, both of which allow us to parse and verify the malware scans on the backend.

Two-factor Authentication

We provide the support needed for testers to secure their accounts using two-factor authentication, which can also be set up as a requirement for customer tests. This helps us ensure that verified tester accounts remain secure.

Customized NDAs

For customers with specific NDA requirements, we support the signature of custom, customer-provided NDA contracts with our testers.

If you have any questions, reach out to us at [email protected].