Rainforest has testing rules every tester is required to follow. As a Rainforest user who writes or reviews test results on a regular basis, it’s important to understand the rules our testing community adheres to so you write tests appropriately and accurately. These rules exist to ensure consistent, high-quality results.
- Rule 1: Pay Close Attention to the Whole Step
- Rule 2: Pay Close Attention to Quotes & Placeholders
- Rule 3: Categorize & Report Pop-Ups & Errors Appropriately
Rule 1: Pay Close Attention to the Whole Step
We set the expectation that you don’t want the tester to rush through instructions or jump to conclusions. The tester must read both the step instruction and question (e.g. “Hit the Login button. Were you successfully logged in?”) before performing any actions.
What this means for you & your test suite
- The instructions you provide must be written accordingly to ensure testers can continue without breaking the rules.
- If you want the tester to remember specific information for a future step, you must make your intention clear in the step instructions where the tester first encounters the information.
For example, “Is the WEIGHT & DIMENSIONS section now displayed on the page? (**please take note of the WEIGHT & DIMENSIONS information. You will need this information in a future step).”
- Never write instructions in a way that forces the tester to answer Yes and agree to skip SEVERAL steps. (e.g. “Answer YES to questions 5-9 and start working on step 10”).
Note: you may request a tester to skip one step at a time, but not for many.
See the tester-facing version of Rule 1 in its entirety and Supporting Examples here.
Rule 2: Pay Close Attention to Quotes & Placeholders
This rule sets expectations around the use of quotes (e.g. “text” or ‘text’) and placeholders (i.e. __) within test instructions.
With minor exceptions, e.g. capitalization, use of quotes in instructions, means that the tester is to check that the text or values within quotes appear exactly as it appears in your application.
Type the book title “Great Gatsby” into the search bar and hit enter on your keyboard.
Are you brought to a page outlining the synopsis for the book?
Testers are trained in the use of an underscore (i.e. _ ) as a placeholder for variable text.
Type “football cleats” in the search bar and hit enter on your keyboard.
Are you directed to a page that says “__ results for football cleats”?
What this means for you & your test suite
If all relevant aspects of quoted text match (i.e. spelling, words, or punctuation) AND testers get the expected outcome of the step question, they must answer YES. Otherwise, they must answer NO. It’s important to note that we instruct testers not to take capitalization differences into account, even when quotes are used.
When to use quotes
- Only use quotes if it’s absolutely necessary for your step instruction, and avoid using them if they’re not critical to the outcome of your test.
- If quotes are used, ensure they are used in a situation where an exact match is possible.
Inappropriate use of quotes:
Look for the “save” icon.
Does clicking on this button enable a pop-up asking to confirm whether or not you’d like to save?”
Where the save button displayed has an image or graphic (e.g. floppy disk icon).
- If you’d like to draw the testers’ attention to a certain word or element on the page and you don't need an exact match, you can use capitalization (e.g. Please find the SEE MORE DETAILS section toward the bottom of the screen.)
Note: You should never use capitalization if what you really require is an exact match. If getting an exact match is your intention, you must use quotes.
How to format quotes
- For consistency, we highly recommend that you use double quotes (“like this”) and stick to just one style. Single quotes (‘like this’) are also acceptable, but less popular. Either way, it’s best to stick to one format throughout a single test for consistency. Our rules for quotes & exact matches apply no matter the format of the quote you choose.
- Typos get the best of us all, and we instruct testers on how to handle these specific situations:
- Mixed quotes: If you accidentally include mixed quotes in your instructions (‘like this”), testers are instructed to assume that your intent was to quote the text and that they should follow the quotes rule accordingly.
- Incomplete quotes: If your instructions include incomplete quotes (“like this), testers are expected to ignore matching since the quotes are incomplete and they can’t guess where the quote might start or end.
- Typos in quoted texted: If testers notice a typo within the quoted portion of your instructions, testers are expected to assume the typo is intended and answer Yes or No depending on whether there’s an exact match between the quoted typo and what they see within the app.
How to use placeholders (i.e. _ ) in your Instructions
- Use the _ placeholder inside or outside of quotes, as well as for an unlimited amount of letters, words, numbers, special characters, etc.
- Use a single underscore (like _ ) or multiple underscores together (like ___ ) to represent one single placeholder, to ensure this is visible.
- Don't use placeholders other than _. If needed, make your intention with the alternative placeholder clear to the tester in instructions before its use.
Example: “In this test, instead of the __ placeholder, we will use X as a placeholder, where X represents the number of items displayed on a page.”
See the tester-facing version of Rule 2 in its entirety and Supporting Examples here.
Rule 3: Categorize & Report Pop-ups & Errors Appropriately
This rule sets expectations around how testers handle certain pop-ups (including errors that come in pop-up format). Rainforest defines a pop-up as anything that suddenly appears (text and/or window) and you don’t need to hover over it in order to see it. We categorize pop-ups into 3 groups:
Group 1: Pop-up which indicates a Bug
This is a pop-up that indicates that something went wrong. Testers are expected to report these pop-ups by answering the step with “No” unless specifically told otherwise in your instructions.
Group 2: Pop-up which doesn't indicate a bug
This is a pop-up that indicates an expected system/browser/application behavior (e.g. a side chat-box, marketing ad, or email subscription offer). Testers are expected to ignore these pop-ups and move on with your test.
Group 3: Pop-up which may indicate a bug, but testers aren't sure if this is the case
If the pop-up doesn’t fall into one of the two other categories (i.e. it’s not an error/crash pop-up, and it’s not an expected behavior by the system/browser/application), and the tester thinks there is a slight chance that this pop-up may indicate a bug, then they are expected to take the most reasonable action to close it, notify you using the SI feature if they choose, and move on with the test if possible.
What this means for you & your test suite
Given the thorough guidelines we provide around pop-ups, there’s usually no need for you to include specific instructions for how testers should interact with pop-ups. However, if you have specific instructions for pop-ups that contradict the main rule, especially if your instructions instruct testers to ignore errors or all pop-ups, you must be very specific in the language you use.
- If you instruct a tester to ignore all pop-ups, they are expected to interpret this as an instruction to ignore all pop-ups, except pop-ups that indicate a bug.
- If you instruct a tester to ignore all pop-ups including error pop-ups, you must provide the expected error in order for testers to ignore it. We strongly encourage resolving errors instead of asking testers to disregard them if possible.
See the tester-facing version of Rule 3 in its entirety and Supporting Examples here.
Suggest Improvement (SI) Feature
You've seen the Suggest Improvement (SI) feature referenced throughout our testing rules.
Testers may use Suggest Improvement if the answer a question as YES. Think of it as “Yes, and … “. They may provide a tip about how your instruction could be phrased better. We encourage testers to use the SI feature in addition to answering the step with a simple“YES” when appropriate.
Testers can use SI to report situations falling under the 5 categories listed below:
- This step can be broken into multiple steps - feedback that the tester is being asked to do too many things in one step, which can lead to confusion
- These instructions are hard to understand - feedback that the test has jargon testers don’t understand or phrasing that could be clearer
- The action you want me to do is not fully described - if instructions omit certain actions that are necessary to continue
- There was an unexpected pop-up in this step - if the tester experiences a pop-up or modal that wasn't accounted for in the instructions that may be benign (see Rule 3) but tester believes you should know about
- I see an additional problem that is not asked as part of the question - the tester noticed something else about your app, e.g. a missing image not mentioned in instructions
While we recommend testers use SI in certain situations, it is never mandatory. If the tester chooses not to use SI, they must follow your instruction to the best of their ability and answer “YES” or “NO” accordingly.
If you’d like to see the tester-facing documentation for the SI feature, along with examples of when testers might choose to use it, you can do so here.