How to use AI Search

Overview

If you need to create test steps that interact with dynamic UI elements, you can use the AI Search action. The AI agent can understand the content on the screen and can interact with your website in ways that aren’t viable with an image-matching approach.

Note: This is an experimental feature and results may be unreliable.

Potential use cases

AI Search is a great way to perform actions that would be difficult to recreate using traditional image-matching steps. Examples include:

• Using the Click action on the latest/first/largest item in a dynamically-changing list
• Using the Click action to add the cheapest/most expensive item to a shopping cart
• Using the Hover action over an element whose appearance tends to change but whose text content remains relatively static.

How it works

Here is an example of how AI Search works using a Click action.

1. Add the Click action
2. Take a screenshot of the closest element to what you are needing to click
3. Wait for the DOM and AI layers to pre-populate
4. Uncheck the Screenshot and DOM layers

3. Describe where you would like the AI agent to click
4. You can preview the step with the virtual machine console open for information on the action and its pass/fail result.

Known limitations

• There are no retries of failed AI steps. The agent looks at the screen once, and if it can’t locate the described element, the step will fail without additional attempts.
• The agent’s thinking phase can take some time, which can range from several seconds to a minute.

Privacy policy

All data and information gathered while using AI actions is sent to OpenAI. The data is retained for 30 days and not used for training. To learn more about OpenAI's policies, click here.

If you have any questions, reach out to us at [email protected].