Visit this article to learn how to generate self-healing tests.

Best practices

The more specific and descriptive the prompt, the more accurate the result.
- E.g., “Enter username and password and then login” is more reliable than “Login."
Avoid prompts that are too high-level — they're generally not specific and descriptive enough.
- E.g., if a user is buying a product, the prompt “buy a product” isn't as reliable as the individual prompts of “select a product," “add it to the cart," “proceed to checkout," “complete the form,” and “make the payment."
You can prompt the AI to generate data by saying something like, “Fill the form using dummy data”.
Clicking on the regenerate icon will regenerate test steps based on the current state of the VM, including whatever page is loaded in the browser. So, when regenerating a block, make sure the state of the VM is the same as it was when the block was first generated. Either (1) manually navigate back to the earlier state or (2) save the test, reset the VM, and preview the test steps up to the relevant block.
Avoid nesting of any kind, including: prompt within a block, prompt within a conditional, conditional within a block, prompt within a snippet, and snippet within a block.
Avoid saving a generative prompt as a snippet.
Note that if you manually add any steps within a generated block or edit any generated steps, your edits will likely be overwritten if the AI decides it needs to update (or self-heal) the relevant block.

Known limitations

Generative self-healing only works on Chrome on Windows 10.
The AI model is largely based on optical character recognition (OCR), so it doesn’t work well on use cases where there aren't textual cues. Poor use cases would include asking the AI to generate test steps based on these elements, for example: form fields without field labels, icons, symbols, and images without text.
Generative self-healing only works if the user has set the number of retries equal to 1.
Generative self-healing doesn't work when blocks contain nested elements, e.g., prompt within a block, prompt within a conditional, conditional within a block, prompt within a snippet, and snippet within a block.
Generating self-healing doesn’t work if you save a prompt as a snippet.
You can't drag-and-drop (i.e., to change the order of) steps or prompts within a test that uses generative AI.
You can't watch a live run of a test using generated blocks (but you can preview).
For self-healed runs, the steps within the automation logs refer to the block numbers, which might not correspond to the actual step number on the block.

FAQs

Test Creation

Is writing a prompt enough to generate steps?

No, you need to click on the generate/regenerate icon.

What’s the difference between preview and generation?

Previewing won't trigger the generation of test steps. It'll simply execute the test steps. During a preview, the “prompt” step will be treated as a comment and will be ignored; only the test steps generated based on the prompt will be executed.

What’s the difference between generation and regeneration?

If a prompt has no underlying steps (for instance, if it’s a new prompt), a magic wand icon is visible. Upon clicking the icon, Rainforest generates the underlying steps for the first time. If a prompt already has underlying steps, a recycle icon is visible. Upon clicking the recycle icon, Rainforest removes the existing generated steps and generates new steps to replace them.

Why is generation slower than preview?

Generation involves making multiple API calls to OpenAI, which uses its model to “think” and generate steps, which could take some time. Preview, on the other hand, is the simple execution of steps within Rainforest (involving no generation), so it’s quicker.

Does self-healing test regeneration automatically happen if a preview fails?

No, a failed preview won't initiate test step regeneration — you'd need to click the recycle icon for the relevant prompt.

Automatic test regeneration (i.e., self-healing) only happens when live test runs fail.

Can a test have multiple generated blocks?

Yes, just not nested within one another.

If I generate a block and am not happy with the generated steps, can I use the regenerate icon?

Yes, provided the state of the site/VM is the same as it was upon initial generation.

If the state of the VM — including, for example, the page loaded in the browser — changed during generation and you wish to regenerate the block, either (1) manually navigate back to the correct state or (2) save the test, reset the VM, and preview the test steps up to the relevant block.

Can I add new steps/edit generated steps within a generated block?

Yes (for now). However, this isn't recommended as there’s a chance your edits will get overridden if the test is self-healed/regenerated during a run.

Can I edit a step within a block and update the screenshot from a past run?

Yes (for now). However, there’s a chance your edits will get overridden if the test is self-healed/regenerated during a run.

Note that Rainforest won't extract the DOM selector for this screenshot, since it wouldn’t have been taken from the live VM. But given that these blocks self-heal,‌ having the DOM selector available probably won't be necessary for avoiding test failures.

Can I manually add steps outside a generated block?

Yes.

Test Runs

Do I need to manually regenerate the blocks after a run to ensure it’s updated/healed?

No, Rainforest will do this automatically.

Will regeneration always be triggered if the test fails in attempt 1?

Regeneration will only be triggered if the test fails on a step within a generated block.

Will self-healing automatically update my tests?

If a test regenerates in response to a failure and passes in its next attempt after regeneration, then Rainforest will automatically update the test to include the new, regenerated steps.

If a test regenerates in response to a failure and doesn't pass its next attempt (i.e., regenerating the test steps did not get the test to a passing state), then, no, Rainforest will not update your test. Its test steps will remain in the same state as before the test run.

Does a test with generated blocks run at the same speed as a test without generated blocks?

If regeneration is not triggered (attempt 2), then yes. If regeneration is triggered, it will be slower.

Which blocks will get triggered for regeneration?

If a test failure occurs outside a generated block, then regeneration isn't triggered.
If a test failure occurs within a generated block:
- None of the blocks before the block that contains the failure are regenerated (since they passed in the first attempt).
- The block containing the failed steps and all subsequent blocks are regenerated.
- For example, if a test has 10 blocks and attempt 1 failed on a step in block 5:
  - Blocks 1-4: not regenerated, run as-is in attempt 2.
  - Blocks 5-10: regenerated and run in attempt 2.

Will manually added/edited steps within a generated block be overridden by regeneration?

Yes.

Will manually added/edited steps outside a generated block be overridden by regeneration?

No.

Will I see a run summary for tests that do not contain generated blocks?

Yes, you'll see summaries when runs pass (including how many steps passed) and fail (including why the tests failed).

Will existing tests be converted to tests with generated blocks?

No, you'll need to do this manually.

If you have any questions, reach out to us at [email protected].