The guidelines and rules tester adhere to when completing tests on Rainforest.
Rainforest QA has a set of testing rules every tester is required to follow when testing your application. As a Rainforest client who writes test cases or reviews test results on a regular basis, it’s important you understand the rules our testing community must adhere to. The rules are as follows:
- Rule 1: Pay Close Attention to the Whole Step
- Rule 2: Pay Close Attention to Quotes & Placeholders
- Rule 3: Categorize & Report Pop-Ups & Errors Appropriately
These rules exist to ensure consistent, high-quality results for you and others. Please read on for a more thorough breakdown of each rule and how it applies to you.
A Note on the Suggest Improvement Feature
You’ll see the Suggest Improvement (SI) feature referenced throughout our testing rules.
Testers may use Suggest Improvement only if the answer to a question if yes — think of it as a “Yes, and … “. They might want to leave you a tip about how your instruction could be phrased better. We encourage testers to use the SI feature in addition to answering the step with “YES”.
Testers can use SI to report situations falling under the 5 categories listed below:
- This step can be broken into multiple steps: you may see this if you are asking them to do too many things in one step, and this is confusing;
- These instructions are hard to understand: you may see this if the testers bumped into some jargon they don’t quite understand, or phrasing that could be clearer;
- The action you want me to do is not fully described: you may see this if your instructions omit certain actions that are too far of a leap for testers to make, or you maybe forgot to add something to make them complete;
- There was an unexpected pop-up in this step: if they bump into a pop-up you didn’t account for in instructions, they think is benign (see Rule 3) but think you should know about it anyway;
- I see an additional problem that is not asked as part of the question: if they notice something else about your app, e.g. a typo in a part not mentioned in instructions.
While we recommend testers use this in certain situations, it is never mandatory. If the tester chooses not to leave a suggestion, they must follow your instruction to the best of their ability and answer “YES” or “NO” accordingly.
Should they choose to use it, we ask that their report includes a polite, helpful comment explaining how the situation could be improved. Unless the tester’s comment was rude or unhelpful, you should never give the tester a poor rating (e.g. 1-2 stars) for using this feature at an appropriate time.
Rule 1: Pay Close Attention to the Whole Step
In this rule, we set the expectation that you don’t want the tester to rush through instructions or jump to conclusions. The tester must read both the step instruction and question (e.g. “Hit the Login button. Were you successfully logged in?” ) before performing any actions. Otherwise, they are at risk of providing inaccurate results, and their work may be flagged for further review by Rainforest QA.
What this means for you and your test suite
Rule 2: Pay Close Attention to Quotes & Placeholders
In this rule, we set expectations around your use of quotes (e.g. “text”) and placeholders (i.e. _ ) within your test instructions.
We tell testers that (with minor exceptions, e.g. capitalization) if you use quotes in your instructions, it means that you want the tester to check that the text or values within quotes appears exactly as it appears in your application. For example, “Type the book title “Great Gatsby” into the search bar and hit enter on your keyboard. Are you brought to a page outlining the synopsis for the book?”
We also explain to testers that when you write test instructions, you are allowed to use an underscore (i.e. _ ) as a placeholder to stand in for variable text that will populate in your application. For example, Type “football cleats” in the search bar and hit enter on your keyboard. Are you directed to a page that says “_ results for football cleats”?
What this means for you & your test suite
How to use quotes in your Instructions
How to use placeholders (i.e. _ ) in your Instructions
- You can use the _ placeholder inside or outside of quotes, as well as for an unlimited amount of letters, words, numbers, special characters, etc.
- You may use a single underscore (like _ ) or multiple underscores stringed together (like ___ ) to represent one single placeholder, to ensure this is visible.
- We also recommend you don't use placeholders other than _ but you may use other text or symbol(s) as placeholders if absolutely necessary. If you choose to do this, make the intention clear in your instructions before you use it. For example, “In this test, instead of the __ placeholder, we will use X as a placeholder, where X is the number of items displayed on a page.”. If you don't, testers are expected to answer with “No” if they encounter instructions using placeholders other than _.
Rule 3: Categorize & Report Pop-ups & Errors Appropriately
In this rule, we set expectations around how testers should handle certain pop-ups (including errors that come in pop-up format). Pop-ups can mean different things to different apps or sites, so it's important to know how testers will interpret them.
Rainforest defines a pop-up as anything that suddenly appears (text and/or window) and you don’t need to hover over it in order to see it.
We categorize pop-ups into 3 groups:
Group 1: Pop-up which indicates a Bug
This is a pop-up that includes an ERROR/CRASH notification. Basically, it indicates that something went wrong. Testers are expected to report these pop-ups with “No” unless specifically told otherwise in your instructions.
Group 2: Pop-up which doesn't indicate a bug
This is a pop-up that indicates an expected system/browser/application behavior (e.g. a side chat-box, marketing ad, or email subscription offer). Testers are expected to ignore these pop-ups and move on with your test.
Group 3: Pop-up which may indicate a bug, but testers aren't sure if this is the case
If the pop-up doesn’t fall into one of the two other categories (i.e. it’s not an ERROR/CRASH pop-up, and it’s not an expected behavior by the system/browser/application), and the tester thinks there is a slight chance that this pop-up may indicate a bug, then they are expected to take the most reasonable action to close it, notify you using the SI feature of they choose, and move on with their test if possible.
What this means for you & your test suite
Given the thorough guidelines we provide around pop-ups, there’s usually no need for you to include specific instructions for how testers should interact with pop-ups. However, if you have specific needs and instructions that contradict the main rule, especially if your instructions instruct testers to ignore errors or all pop-ups, you must be very specific in the language you use.
- If you instruct a tester to ignore all pop-ups, they are expected to interpret this as an instruction to ignore all pop-ups, except ERROR/CRASH notifications.
- If you instruct a tester to ignore all pop-ups including error pop-ups, they are expected to ignore all pop-ups, including ERROR/CRASH notifications. Be very careful with this - a poorly set up or buggy testing environment can lead to frequent failures and confusion, since testers are on the lookout for bugs. Even so, if a tester encounters an ERROR/CRASH notification they think you should know about despite your warnings, they are allowed to alert you using the SI category “There was an unexpected popup in this step”, and then continue with the test if possible.
- If you want the tester to ignore a specific type of ERROR/CRASH pop-up, then you must clearly state this in the instructions along with a description of the specific ERROR/CRASH pop-up they should ignore. Otherwise, testers will report the pop-up with “NO”.
There you have it!
For further details about how testers will handle pop-ups that appear during testing, please see our documentation on how to test with pop-ups.
If you have any questions about the rules testers are supposed to follow, please let our team know at firstname.lastname@example.org!