Troubleshooting Native Mobile Tests

This document lists some common problems and errors that can occur when running mobile tests and how to troubleshoot them.

Introduction

It's possible to test mobile apps using real devices (via AWS Device Farm) or our mobile VMs. In both cases, customers should create for each app an App URL (Settings > Sites > Add App) pointing to the address from where the app will be downloaded.

The App URL should lead to a direct download, which means it should not redirect, require credentials or have access limited to some network or IP.

At the beginning of each Run, the app will be download from the specified App URL and automatically installed on the real devices or VMs. The Run will be errored if for any reason the download or the app installation fails.

Also, the apps should be correctly build and configured to run on the real devices or VMs. For more information, please check the following support docs:

Common Error Messages

Runs failing with "Couldn't download file…"

This error occurs when the application fails to download from the provided app URL.
To resolve this, follow the below steps

  1. Check if the test is pointing to an App URL and not a Site URL on all environments.
  2. Check if you can download the app using only the specified App URL without a redirect, providing any credentials or having to click on any link/button to start the download.
  3. Apps hosted on Dropbox should specify dl=1 as a query parameter on the App URL to force the app to download

Runs failing with "…app failed to communicate with our platform."

This error message indicative that the Rainforest framework was incorrectly added to to the app's .IPA file. To resolve this:

  1. Check with the customer if he have added and started the Rainforest.framework on the specified iOS app.
  2. Instructions to add the Rainforest.framework can be found here
  3. Confirm that the customer have add the Rainforest.framework by looking for the following line in the Device Logs:  "RF: Loading Rainforest framework - v 1.1.1"
  4. If above line cannot be located, this indicates that the Rainforest framework was not correctly added to the app. 

Runs failing with "Please check that your app is configured properly."

This error usually occurs when it was not possible to install or launch the app on the real device or VM.

  1. First, check the App/Device Logs.
  2. Check with the customer if he have provided a valid app file:

iOS real devices:

  • Should be an .ipa file with the Rainforest.framework included.
  • build with a valid "Minimum OS Version".
  • Apps build with iOS 10.0 as the "Minimum OS Version" can't run on iOS 9 devices!!

iOS VMs

  • Should be an .zip file with the .app file included.
  • Should be compiled for x86.
  • The following line in the App Logs indicates that an app was not build to run in the simulators: Program specified by service does not contain one of the requested architectures
  • When creating the .app file the user must inform the -sdk iphonesimulator param in the xcodebuild command.
  • Users can test the .app file locally by following the troubleshooting instructions provided here.

Android VMs

  • Should be an .apk file.
  • Should be compiled to run on x86.

Android real devices:

  • Should be an .apk file.
  • Should be compiled to run on x86 and ARM.
  • Samsung Galaxy Tab 3 10.1" runs on x86!

Runs failing with "There are not available xxx devices at the moment…"

This error means that at this moment there are not available phones of the requested type due to maintenance issues on AWS Device Farm. It can also shows up when the app was not build to support the device's OS version.

  1. Verify that it is possible to reproduce the same error on preview. If this can be reproduced, this means that this device is on maintenance on Device Farm.
  2. When this occurs let us at support@rainforestqa.com or through Intercom so that we can monitor this. Usually, AWS will make the device available again after some hours.
  3. If the preview works, please check that the app was built to run on the device's OS version. It is particular important to check the app's "Minimum OS Version".

Testers reporting black or frozen terminals on iOS real devices

This problem can happen when the app crashes in the middle of the test or because a problem on our script to dismiss permission dialogs.

  1. First check the Device Logs; usually the error/crash will be in the second log file.
  2. Check the device video. In the moment where the testers are reporting the problem. If the app opens Safari or any other external application, please change the test as testers can only work within the app in a sand.
  3. If in the video a system permission dialog appears, please let us know at support@rainforestqa.com and we will investigate with our engineering team.

Browser loaded instead of the App in the beginning of the test

If and when the testers report that the browser was showed instead of the app this usually means that the app crashed right after be launched. If this occurs, please check the App Logs for the error event and watch the tester video.

App not installed on Android real devices

On Android tests using real phones (via Device Farm) the app will not be installed if it does not support the Android version installed on the devices.

  1. Ask the customer to check the App Logs + Videos.
  2. Usually, the following error message will appears in the logs when the app requires a newer Android version: "Requires newer sdk version #XX (current version is #YY)"

Error adding the Rainforest framework

Dyld Message: Library not loaded: @rpath/Rainforest.framework/Rainforest

The error above happens when the Rainforest.framework in not added as an embedded binary in the Xcode project.

  1. Check that the Rainforest framework has been added as an "Embedded Binary" in this app/project.

If you encounter error that is not detailed above, please let us know at support@rainforestqa.com. 

Did this answer your question?