How to use AI Assertions

Overview

If you need to confirm whether or not some text is visible on the screen, you can use the AI Assertion action to answer yes/no questions. It has an understanding of the content of the visible page and can even answer questions about dynamic content that changes between runs.

🚧

This is an experimental feature that’s currently included in your plan for free. We’re evaluating how this action is used and if it will need to come at an additional cost.

Potential use cases

AI Assertions are a great way to perform human-like evaluations that would be difficult to recreate using traditional assertion steps. Examples include:

• The AI agent has access to the test data, so you can ask, “Do you see {{ login_data.email }} in the email field?” and it will know to look for the exact email address used in your test.
• The AI agent also knows the current date, so you can ask, “Is the delivery date set to the upcoming Monday?” and it will be able to answer this correctly.
• Because the agent can understand the context of the content on the page, it can accurately answer broader questions like, “Is the homepage visible?”

Note: To verify some text is not visible on the page, frame the question so the answer to some text not being visible would be “yes”. E.g., “Is the login button not visible?”

How it works

Here is an example of using AI Assertion to see if something is visible:

1. Type a yes/no question into the step

   

2. Preview the step to confirm its expected behavior

3. Each answer will come back with a justification in the Virtual Machine Console for whether the step passed (the answer was “yes”) or failed (the answer was “no”)

   

Rules AI Assertion follows

The AI agent is configured to follow these rules:

• If the question is not formulated as a yes/no question, explain that to the user and answer “no”
• If the question requires information that you don't have (e.g., colors or graphics), explain that to the user and answer “no”
• If the question is about sorting, assume non-strict sorting (duplicates are okay).
• If there are multiple questions, all of them have to be answered by “yes” for the answer to be “yes”.
• The text for the question is extracted by our optical character recognition system (OCR), so expect and ignore small errors typical to OCR software.

Note: To verify something is not visible on the page, frame the question so the answer to something not being visible would be “yes”. E.g., “Is the login button not visible?”

Known limitations

There’s a couple of limitations today:

• There are no retries of failed AI steps. The agent looks at the screen once, and if it can’t locate the described element, the step will fail without additional attempts.
• Today, only text content is taken into account, so questions about colors or images will all result in a failure.
• The agent’s thinking phase can take some time, which can range from several seconds to a minute.

Privacy policy

All data and information gathered while using AI actions is sent to OpenAI. The data is retained for 30 days and not used for training. To learn more about OpenAI's policies, click here.

If you have any questions, reach out to us at [email protected].