How to use AI Search

Overview

If you need to create test steps that interact with dynamic UI elements, you can use the AI Search action. The AI can understand the text visible on the screen and can interact with your website in ways that aren’t viable with an image-matching approach.

Note: This is an experimental feature and results may be unreliable.

🚧

This is an experimental feature that’s currently included in your plan for free. We’re evaluating how this action is used and if it will need to come at an additional cost.

Potential use cases

AI Search is a great way to perform actions that would be difficult to recreate using traditional image-matching steps. Examples include:

• Using the Click action on the latest/first/largest item in a dynamically-changing list
• Using the Click action to add the cheapest/most expensive item to a shopping cart
• Using the Hover action over an element whose appearance tends to change but whose text content remains relatively static.

How it works

Here is an example of how AI Search works using a Click action.

1. Add the Click action

2. Toggle to select AI Search

3. Describe where you would like the AI agent to click

📘

Note: The AI agent can only see and interact with text. Any reference to images or color will result in a failed step.

4. You can preview the step with the virtual machine console open for information on the action and its pass/fail result.

Known limitations

• There are no retries of failed AI steps. The agent looks at the screen once, and if it can’t locate the described element, the step will fail without additional attempts.
• Today, only text content is taken into account, so questions about colors or images will all result in a failure.
• The agent’s thinking phase can take some time, which can range from several seconds to a minute.

Privacy policy

All data and information gathered while using AI actions is sent to OpenAI. The data is retained for 30 days and not used for training. To learn more about OpenAI's policies, click here.

---

If you have any questions, reach out to us at [email protected].