Rainforest QA has a set of testing rules every tester is required to follow when testing your application. As a Rainforest user who writes or reviews test on a regular basis, it’s important you understand the rules our testing community adheres to. The rules are as follows:
- Rule 1: Pay Close Attention to the Whole Step
- Rule 2: Pay Close Attention to Quotes & Placeholders
- Rule 3: Categorize & Report Pop-Ups & Errors Appropriately
These rules exist to ensure consistent, high-quality results.
Suggest Improvement (SI) Feature
You’ll see the Suggest Improvement (SI) feature referenced throughout our testing rules.
Testers may use Suggest Improvement only if the answer to a question is YES. Think of it as “Yes, and … “. They may provide a tip about how your instruction could be phrased better. We encourage testers to use the SI feature in addition to answering the step with a simple“YES” when appropriate.
Testers can use SI to report situations falling under the 5 categories listed below:
- This step can be broken into multiple steps
may see if being asked to do too many things in one step, as this is confusing
- These instructions are hard to understand
may see if test has jargon they don’t understand, or phrasing that could be clearer
- The action you want me to do is not fully described
if instructions omit certain actions that are too far of a leap for testers to make, or test is missing a step to make it complete
- There was an unexpected pop-up in this step
if tester experiences a pop-up or modal that wasn't accounted for in the instructions that may be benign (see Rule 3) but tester believes you should know about
- I see an additional problem that is not asked as part of the question
notice something else about your app, e.g. a typo in a part not mentioned in instructions
While we recommend testers use SI in certain situations, it is never mandatory. If the tester chooses not to use SI, they must follow your instruction to the best of their ability and answer “YES” or “NO” accordingly.
Should the tester choose to use the SI feature, we ask that their report includes a polite, helpful comment explaining how the situation could be improved. Unless the tester’s comment was rude or unhelpful, you should not give the tester a poor rating (e.g. thumbs-down) for using the SI feature at an appropriate time.
If you’d like to see the tester-facing documentation for the SI feature, along with examples of when testers might choose to use it, you can do so here.
Rule 1: Pay Close Attention to the Whole Step
In this rule, we set the expectation that you don’t want the tester to rush through instructions or jump to conclusions. The tester must read both the step instruction and question (e.g. “Hit the Login button. Were you successfully logged in?” ) before performing any actions. Otherwise, they are at risk of providing inaccurate results, and their work may be flagged for further review by Rainforest QA.
What this means for you & your test suite
- The instructions you provide the tester must be written clearly & in accordance with the testing rules they are required to follow, and must be accompanied by a single YES/NO question per step.
- Note: Testers are only allowed to answer YES if they fully understand the instructions and the question provided, they followed those instructions exactly, and the answer to the question is YES. Otherwise, the tester must click NO. Unless the tester’s bug report was rude or unhelpful, you should not give the tester a poor rating (e.g. thumbs-down) if they submitted unsatisfactory results that can be traced to poorly written instructions.
- If you want the tester to remember some specific information for a future step, you must make your intention clear in the step instructions where the tester first encounters the information. For example, “Click the VIEW DETAILS button. Is the WEIGHT & DIMENSIONS section now displayed on the page? (**please take note of the information you see under the WEIGHT & DIMENSIONS section. You will need to recall this information in a future step).”
- You should never write instructions in a way that forces the tester to answer Yes and agree to skip several steps without executing them in order to reach a later one (e.g. “Answer YES to questions 5-9 and start working from 10”).
- Rainforest advises testers to immediately report a job with NO should they encounter a step with instructions of this nature. Unless the tester’s comment was rude or unhelpful, you should never give the tester a poor rating (e.g. thumbs-down) for answering NO to this type of instruction.
- Note: it is acceptable to request a tester do this from one step to another, but not for many. E.g. “You may or may not encounter an pop-up on this step. Continue to the next step if you do.” is an example of an acceptable step instruction.
See tester-facing version of Rule 1 in it’s entirety and Supporting Examples here.
Rule 2: Pay Close Attention to Quotes & Placeholders
In this rule, we set expectations around your use of quotes (e.g. “text” or ‘text’) and placeholders (i.e. __ ) within your test instructions.
We tell testers that (with minor exceptions, e.g. capitalization) if you use quotes in your instructions, it means that you want the tester to check that the text or values within quotes appears exactly as it appears in your application. For example, “Type the book title “Great Gatsby” into the search bar and hit enter on your keyboard. Are you brought to a page outlining the synopsis for the book?”
We also explain to testers that when you write test instructions, you are allowed to use an underscore (i.e. _ ) as a placeholder to stand in for variable text that will populate in your application. For example, Type “football cleats” in the search bar and hit enter on your keyboard. Are you directed to a page that says “__ results for football cleats”?
What this means for you & your test suite
If all relevant aspects of quoted text match (i.e. spelling, words, or punctuation) AND testers get the expected outcome of the step question, they must answer YES. Otherwise, they must answer NO. It’s important to note that we instruct testers not to take capitalization differences into account, even when quotes are used. You should never give a tester a poor rating (e.g. thumbs-down) for not failing a step over capitalization differences.
When to use quotes
- You should only use quotes if it’s absolutely necessary for your step instruction, and avoid using them if they’re not critical to the outcome of your test.
- If you choose to use quotes, make sure you are using them in a situation where an exact match is possible. Here’s an example of when it’s appropriate to use quotes vs. when it is not:
- Appropriate use of quotes: “Look for the “save” button. Does clicking on this button enable a pop-up asking to confirm whether or not you’d like to save?” where the button displayed in your application actually has the text “save” on it.
- Inappropriate use of quotes: “Look for the “save” button. Does clicking on this button enable a pop-up asking to confirm whether or not you’d like to save?” where the save button displayed in your application does not have the text “save” written on it, but is instead another image or graphic (e.g. floppy disk icon). In situations like these, we advise testers to answer “Yes” if it’s obvious to them what you mean and report the inappropriate use of quotes to you using the SI feature.
- If you’d like to draw the testers’ attention to a certain word or element on the page, but don’t care about an exact match, you can use CAPS or asterisks (e.g. “Please find the SEE MORE DETAILS section toward the button of the screen and click the EXPAND button. Did the section expand to display more details?”)
- Important Note: You should never use CAPS or asterisks if what you really require is an exact match. If getting an exact match is your intention, you must use quotes.
How to format quotes
- For consistency, we highly recommend that you only use double quotes (“like this”) and stick to just one style. Single quotes (‘like this’) or even the character for grave accent (
like this) are also acceptable, but less popular. Either way, it’s best stick to one format throughout a single test for consistency. Our rules for quotes & exact matches apply no matter the format of quote you choose.
- Typos get the best of us all, and we instruct testers on how to handle these specific situations:
- Mixed quotes: If you accidentally include mixed quotes in your instructions (‘like this”), testers are instructed to assume that your intent was to quote the text and that they should follow the quotes rule accordingly. This ‘mixed quotes” scenario is also a time where we encourage testers to use the SI category “I see an additional problem that is not asked as part of the question” and report the typo to you to reduce confusion.
- Incomplete quotes: If your instructions include incomplete quotes (“like this), testers are expected to ignore matching since the quotes are incomplete and they can’t guess where the quote might start or end, but leave a comment using the SI category “These instructions are hard to understand” to point out the incomplete quote in case an exact match was your intent.
- Typos in quoted texted: If testers notice a typo within the quoted portion of your instructions, testers are expected to assume the typo is intended, and answer Yes or No depending on whether there’s an exact match between the quoted typo and what they see within the app. If you make a typo in the unquoted portion of your instructions, we encourage testers to report this to you using the SI category “I see an additional problem that is not asked as part of the question.”
How to use placeholders (i.e. _ ) in your Instructions
- You can use the _ placeholder inside or outside of quotes, as well as for an unlimited amount of letters, words, numbers, special characters, etc.
- You may use a single underscore (like _ ) or multiple underscores stringed together (like ___ ) to represent one single placeholder, to ensure this is visible.
- We also recommend you don't use placeholders other than _ but you may use other text or symbol(s) as placeholders if absolutely necessary. If you choose to do this:
- You must make your intention with the alternative placeholder clear to the tester in your instructions before you use it. For example, “In this test, instead of the __ placeholder, we will use X as a placeholder, where X represents the number of items displayed on a page.”
- If you do not clearly explain your use of the alternative placeholder in your test instructions, testers are expected to answer with “No” if they encounter instructions using placeholders other than __.
See tester-facing version of Rule 2 in it’s entirety and Supporting Examples here.
Rule 3: Categorize & Report Pop-ups & Errors Appropriately
In this rule, we set expectations around how testers should handle certain pop-ups (including errors that come in pop-up format). Rainforest defines a pop-up as anything that suddenly appears (text and/or window) and you don’t need to hover over it in order to see it. We categorize pop-ups into 3 groups:
Group 1: Pop-up which indicates a Bug
This is a pop-up that includes an ERROR/CRASH notification. Basically, it indicates that something went wrong. Testers are expected to report these pop-ups with “No” unless specifically told otherwise in your instructions.
Group 2: Pop-up which doesn't indicate a bug
This is a pop-up that indicates an expected system/browser/application behavior (e.g. a side chat-box, marketing ad, or email subscription offer). Testers are expected to ignore these pop-ups and move on with your test.
Group 3: Pop-up which may indicate a bug, but testers aren't sure if this is the case
If the pop-up doesn’t fall into one of the two other categories (i.e. it’s not an ERROR/CRASH pop-up, and it’s not an expected behavior by the system/browser/application), and the tester thinks there is a slight chance that this pop-up may indicate a bug, then they are expected to take the most reasonable action to close it, notify you using the SI feature of they choose, and move on with their test if possible.
What this means for you & your test suite
Given the thorough guidelines we provide around pop-ups, there’s usually no need for you to include specific instructions for how testers should interact with pop-ups. However, if you have specific instructions for pop-ups that contradict the main rule, especially if your instructions instruct testers to ignore errors or all pop-ups, you must be very specific in the language you use.
- If you instruct a tester to ignore all pop-ups, they are expected to interpret this as an instruction to ignore all pop-ups, except ERROR/CRASH notifications.
- If you instruct a tester to ignore all pop-ups including error pop-ups, they are expected to ignore all pop-ups, including ERROR/CRASH notifications. However, if a tester encounters an ERROR/CRASH notification they think you should still know about, they are allowed to alert you using the SI category “There was an unexpected popup in this step”, and then continue with the test if possible.
- If you want the tester to ignore a specific type of ERROR/CRASH pop-up, then you must clearly state this in the instructions along with a description of the specific ERROR/CRASH pop-up they should ignore. Otherwise, testers will report the pop-up with “NO”.
See tester-facing version of Rule 3 in it’s entirety and Supporting Examples here.