How to use AI Search

Overview

If you need to create test steps that interact with dynamic UI elements, you can use the AI Search action. The AI agent can understand the content on the screen and can interact with your website in ways that aren’t viable with an image-matching approach.

Note: This is an experimental feature and results may be unreliable.

🚧
This is an experimental feature that’s currently included in your plan for free. We’re evaluating how this action is used and if it will need to come at an additional cost.

Potential use cases

AI Search is a great way to perform actions that would be difficult to recreate using traditional image-matching steps. Examples include:

Using the Click action on the latest/first/largest item in a dynamically-changing list
Using the Click action to add the cheapest/most expensive item to a shopping cart
Using the Hover action over an element whose appearance tends to change but whose text content remains relatively static.

How it works

Here is an example of how AI Search works using a Click action.

Add the Click action
Take a screenshot of the closest element to what you are needing to click
Wait for the DOM and AI layers to pre-populate
Uncheck the Screenshot and DOM layers

Describe where you would like the AI agent to click
You can preview the step with the virtual machine console open for information on the action and its pass/fail result.

Known limitations

There are no retries of failed AI steps. The agent looks at the screen once, and if it can’t locate the described element, the step will fail without additional attempts.
The agent’s thinking phase can take some time, which can range from several seconds to a minute.

Privacy policy

All data and information gathered while using AI actions is sent to OpenAI. The data is retained for 30 days and not used for training. To learn more about OpenAI's policies, click here.

If you have any questions, reach out to us at [email protected].