Visit this article to learn how to generate self-healing tests.

Best practices

Write your prompts to capture the intent behind what you’re testing.

Rainforest AI blocks can help you avoid maintenance when your application's UX changes, while the user flow and underlying intention remain the same. Think of your instructions to the AI agents as if you're guiding a QA tester without prior knowledge of your application's purpose or functionality.

When your test fails and attempts to self-heal, we'll revert to the original prompt you wrote. If that prompt accurately reflects what you intend to test, you'll avoid unnecessary maintenance and falsely passing tests.

For example:

You could write a prompt that instructs the user to

“create an account using dummy data” or
“Enter dummy data and accept the terms of service to create an account”

The additional detail you chose to include in your prompt means that if the option to accept terms of service is removed, the outcome of the test will change.

If a term of service checkbox is present, and you-

Wrote prompt 1: Then your test will self-heal based on prompt 1. Given the intent behind this prompt, you won’t need to update the test yourself because it will pass via self-healing.
Wrote prompt 2: We fall back to the prompt that does include the instruction to accept the terms of service. Because the checkbox no longer exists, we won’t be able to self-heal. Given the intent behind your prompt, this is now flagged as a failure and can be treated as a bug.

The more specific and descriptive the prompt, the more accurate the result.

You can use high-level prompts, but generating the steps you expect may take a couple of tries. AI is a relatively new feature and can complete some high-level tasks well, but it hasn’t been instructed on your application. The more detailed instructions you give it, the more likely it’ll generate the appropriate steps on the first try.

For example:

If a user is buying a product, the prompt “buy a product” isn't as reliable as the individual prompts of “select a product," “add it to the cart," “proceed to checkout," “complete the form,” and “make the payment."

Create prompts and tests that are deterministic wherever possible.

Be careful when covering scenarios that are dynamic (i.e. they’ll change each time the test is run). You should avoid writing tests designed in a way that will require self-healing every time they’re run. Doing so will create slower and flakier tests.

AI does add some flexibility to testing that makes it easier to cover scenarios that aren’t 100% predictable, but you should make your flows deterministic and design them to pass without self-healing whenever possible. Writing a prompt with language like “any” or “one of” might generate the right steps on the first try, but it will lead to slower test runs and can lead to test flakiness down the line.

We’re working on making AI blocks “smarter” about generating more dynamic steps (e.g. AI search), but there are other Rainforest tools that might be better to handle these scenarios that we don’t cover yet that you could write with manual steps (e.g., conditionals or masking).

For example:

If you write a prompt to instruct the user to add “any product” to the cart but the order that those products displayed on your site changes every time the page is loaded, the test will fail and have to be self-healed every time that it’s run. Falling back to AI will take longer, and occasionally might not heal as expected (no AI product gets it right 100% of the time).

Two options would make this test more reliable:

You could add an instruction to search for a specific product or keyword first so that the same product (or a few similar products) is always displayed and can be selected. That way the test will only fail and fall back to self-healing if that product is no longer available.
Instead of using an AI block to cover this whole flow, you could add a manual step to select the first item in the list by masking out the product details and then creating an AI block to go through the add-to-cart and checkout flows.

Use test data:

You can prompt the AI to generate data by saying something like, “Fill the form using dummy data” or “Fill the form using {{logins.firstname}} and {{logins.email}}” if you’d like to reference either random Rainforest test data or test data that you’ve uploaded yourself.

Edit your prompt and regenerate the steps when a flow changes:

Note that if you manually add any steps within a generated block or edit any generated steps, your edits will likely be overwritten if the AI decides it needs to update (or self-heal) the relevant block. The best practice is to update your prompt and regenerate those steps so you know that it will self-heal properly if small changes are made to your application.

Known limitations

Generative self-healing only works on Chromium-based browsers on Windows 10.
Generative self-healing only works if the user has set the number of retries equal to 1.
Generating AI blocks when multiple tabs/windows are open is an experimental feature.
Currently not supported by the AI agent: AI Assertion, conditional block, drag and drop, repeated actions, copying and pasting, masking, text matching

FAQs

Test Creation

Why is generation slower than preview?

Generation involves making multiple API calls to OpenAI, which uses its model to “think” and generate steps, which could take some time. Preview, on the other hand, is the simple execution of steps within Rainforest (involving no generation), so it’s quicker.

Does self-healing test regeneration automatically happen if a preview fails?

No, a failed preview won't initiate test step regeneration — you'd need to click the regenerate icon for the relevant prompt.

Automatic test regeneration (i.e., self-healing) only happens when live test runs fail.

Can a test have multiple generated blocks?

Yes, just not nested within one another. N.B. that we’ll only heal on block in a test so if there are multiple failures within a test, you’ll need to review it.

If I generate a block and am not happy with the generated steps, can I use the regenerate icon?

Yes, provided the state of the site/VM is the same as it was upon initial generation.

If the state of the VM — including, for example, the page loaded in the browser — changed during generation and you wish to regenerate the block, either (1) manually navigate back to the correct state or (2) save the test, reset the VM, and preview the test steps up to the relevant block.

Can I add new steps/edit generated steps within a generated block?

Yes (for now). However, this isn't recommended as there’s a chance your edits will get overridden if the test is self-healed/regenerated during a run.

Test Runs

Do I need to manually regenerate the blocks after a run to ensure it’s updated/healed?

No, Rainforest will do this automatically.

Will regeneration always be triggered if the test fails in attempt 1?

Regeneration will only be triggered if the test fails on a step within a generated block.

Will self-healing automatically update my tests?

If a test regenerates in response to a failure and passes in its next attempt after regeneration, then Rainforest will automatically update the test to include the new, regenerated steps.

If a test regenerates in response to a failure and doesn't pass its next attempt (i.e., regenerating the test steps did not get the test to a passing state), then, no, Rainforest will not update your test. Its test steps will remain in the same state as before the test run.

Does a test with generated blocks run at the same speed as a test without generated blocks?

If regeneration is not triggered (attempt 2), then yes. If regeneration is triggered, it will be slower.

Which blocks will get triggered for regeneration?

Only the block that contains the failure will be regenerated.

If you have any questions, reach out to us at [email protected].