- NAME
-
- gcloud firebase test android run - invoke a test in Firebase Test Lab for Android and view test results
- SYNOPSIS
-
-
gcloud firebase test android run
[ARGSPEC
] [--app
=APP
] [--device
=DIMENSION
=VALUE
,[…]] [--test
=TEST
] [--timeout
=TIMEOUT
] [--type
=TYPE
] [--additional-apks
=APK
,[APK
,…]] [--app-package
=APP_PACKAGE
] [--async
] [--auto-google-login
] [--client-details
=[KEY
=VALUE
,…]] [--directories-to-pull
=[DIR_TO_PULL
,…]] [--environment-variables
=[KEY
=VALUE
,…]] [--network-profile
=PROFILE_ID
] [--num-flaky-test-attempts
=int
] [--obb-files
=OBB_FILE
,[OBB_FILE
]] [--other-files
=DEVICE_PATH
=FILE_PATH
,[…]] [--performance-metrics
] [--record-video
] [--results-bucket
=RESULTS_BUCKET
] [--results-dir
=RESULTS_DIR
] [--results-history-name
=RESULTS_HISTORY_NAME
] [--scenario-labels
=LABEL
,[LABEL
,…]] [--scenario-numbers
=int
,[int
,…]] [--test-package
=TEST_PACKAGE
] [--test-runner-class
=TEST_RUNNER_CLASS
] [--test-targets
=TEST_TARGET
,[TEST_TARGET
,…]] [--use-orchestrator
] [--resign
] [--robo-directives
=[TYPE
:RESOURCE_NAME
=INPUT
,…]] [--robo-script
=ROBO_SCRIPT
] [--device-ids
=MODEL_ID
,[MODEL_ID
,…],-d
MODEL_ID
,[MODEL_ID
,…]] [--locales
=LOCALE
,[LOCALE
,…],-l
LOCALE
,[LOCALE
,…]] [--orientations
=ORIENTATION
,[ORIENTATION
],-o
ORIENTATION
,[ORIENTATION
]] [--os-version-ids
=OS_VERSION_ID
,[…],-v
OS_VERSION_ID
,[…]] [--filter
=EXPRESSION
] [--limit
=LIMIT
] [--page-size
=PAGE_SIZE
] [--sort-by
=[FIELD
,…]] [GCLOUD_WIDE_FLAG …
]
-
- DESCRIPTION
-
gcloud firebase test android run
invokes and monitors tests in Firebase Test Lab for Android.Three main types of Android tests are currently supported:
-
robo
: runs a smart, automated exploration of the activities in your Android app which records any installation failures or crashes and builds an activity map with associated screenshots and video. -
instrumentation
: runs automated unit or integration tests written using a testing framework. Firebase Test Lab for Android currently supports the Espresso and UI Automator 2.0 testing frameworks. -
game-loop
: executes a special intent built into the game app (a "demo mode") that simulates the actions of a real player. This test type can include multiple game loops (also called "scenarios"), which can be logically organized using scenario labels so that you can run related loops together. Refer to https://firebase.google.com/docs/test-lab/android/game-loop for more information about how to build and run Game Loop tests.
The type of test to run can be specified with the
--type
flag, although the type can often be inferred from other flags. Specifically, if the--test
flag is present, the test--type
defaults toinstrumentation
. If--test
is not present, then--type
defaults torobo
.All arguments for
gcloud firebase test android run
may be specified on the command line and/or within an argument file. Run$ gcloud topic arg-files
for more information about argument files. -
- EXAMPLES
-
To invoke a robo test lasting 100 seconds against the default device
environment, run:
gcloud firebase test android run --app=APP_APK --timeout=100s
When specifying devices to test against, the preferred method is to use the --device flag. For example, to invoke a robo test against a virtual, generic MDPI Nexus device in landscape orientation, run:
gcloud firebase test android run --app=APP_APK --device=model=NexusLowRes,orientation=landscape
To invoke an instrumentation test against a physical Nexus 6 device (MODEL_ID: shamu) which is running Android API level 21 in French, run:
gcloud firebase test android run --app=APP_APK --test=TEST_APK --device=model=shamu,version=21,locale=fr
To test against multiple devices, specify --device more than once:
gcloud firebase test android run --app=APP_APK --test=TEST_APK --device=model=Nexus4,version=19 --device=model=Nexus4,version=21 --device=model=NexusLowRes,version=25
To invoke a robo test on an Android App Bundle, pass the .aab file using the --app flag.
gcloud firebase test android run --app=bundle.aab
You may also use the legacy dimension flags (deprecated) to specify which devices to use. Firebase Test Lab will run tests against every possible combination of the listed device dimensions. Note that some combinations of device models and OS versions may not be valid or available in Test Lab. Any unsupported combinations of dimensions in the test matrix will be skipped.
For example, to execute a series of 5-minute robo tests against a very comprehensive matrix of virtual and physical devices, OS versions, locales and orientations, run:
gcloud firebase test android run --app=APP_APK --timeout=5m --device-ids=shamu,NexusLowRes,Nexus5,g3,zeroflte --os-version-ids=19,21,22,23,24,25 --locales=en_GB,es,fr,ru,zh --orientations=portrait,landscape
The above command will generate a test matrix with a total of 300 test executions, but only the subset of executions with valid dimension combinations will actually run your tests.
To help you identify and locate your test matrix in the Firebase console, run:
gcloud firebase test android run --app=APP_APK --client-details=matrixLabel="Example matrix label"
Controlling Results Storage
By default, Firebase Test Lab stores detailed test results for a limited time in a Google Cloud Storage bucket provided for you at no charge. If you wish to use a storage bucket that you control, or if you need to retain detailed test results for a longer period, use the
--results-bucket
option. See https://firebase.google.com/docs/test-lab/analyzing-results#detailed for more information.Detailed test result files are prefixed by default with a timestamp and a random character string. If you require a predictable path where detailed test results are stored within the results bucket (say, if you have a Continuous Integration system which does custom post-processing of test result artifacts), use the
--results-dir
option.Note that each test invocation
. Possible strategies could include using a UUID or sequence number formust
have a unique storage location, so never reuse the same value for--results-dir
between different test runs--results-dir
.For example, to run a robo test using a specific Google Cloud Storage location to hold the raw test results, run:
gcloud firebase test android run --app=APP_APK --results-bucket=gs://my-bucket --results-dir=my/test/results/<unique-value>
To run an instrumentation test and specify a custom name under which the history of your tests will be collected and displayed in the Firebase console, run:
gcloud firebase test android run --app=APP_APK --test=TEST_APK --results-history-name='Excelsior App Test History'
Argument Files
All test arguments for a given test may alternatively be stored in an argument group within a YAML-formatted argument file. The
ARG_FILE
may contain one or more named argument groups, and argument groups may be combined using theinclude:
attribute (Run$ gcloud topic arg-files
for more information). The ARG_FILE can easily be shared with colleagues or placed under source control to ensure consistent test executions.To run a test using arguments loaded from an ARG_FILE named
excelsior_args
, which contains an argument group namedrobo-args:
, use the following syntax:gcloud firebase test android run path/to/excelsior_args:robo-args
- POSITIONAL ARGUMENTS
-
- [
ARGSPEC
] -
An ARG_FILE:ARG_GROUP_NAME pair, where ARG_FILE is the path to a file containing
groups of test arguments in yaml format, and ARG_GROUP_NAME is the particular
yaml object holding a group of arg:value pairs to use. Run
$ gcloud topic arg-files
for more information and examples.
- [
- COMMONLY USED FLAGS
-
--app
=APP
- The path to the application binary file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. Android App Bundles are specified as .aab, all other files are assumed to be APKs.
--device
=DIMENSION
=VALUE
,[…]-
A list of
pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are:DIMENSION=VALUE
model
,version
,locale
, andorientation
. If any dimensions are omitted, they will use a default value. The default value, and all possible values, for each dimension can be found with the
command for that dimension, such aslist
$ gcloud firebase test android models list
.--device
is now the preferred way to specify test devices and may not be used in conjunction with--devices-ids
,--os-version-ids
,--locales
, or--orientations
. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.Examples:
--device model=Nexus6 --device version=23,orientation=portrait --device model=shamu,version=22,locale=zh_CN,orientation=default
--test
=TEST
-
The path to the binary file containing instrumentation tests. The given path may
be in the local filesystem or in Google Cloud Storage using a URL beginning with
gs://
. --timeout
=TIMEOUT
-
The max time this test execution can run before it is cancelled (default: 15m).
It does not include any time necessary to prepare and clean up the target
device. The maximum possible testing time is 45m on physical devices and 60m on
virtual devices. The
TIMEOUT
units can be h, m, or s. If no unit is given, seconds are assumed. Examples:-
--timeout 1h
is 1 hour -
--timeout 5m
is 5 minutes -
--timeout 200s
is 200 seconds -
--timeout 100
is 100 seconds
-
--type
=TYPE
-
The type of test to run.
TYPE
must be one of:instrumentation
,robo
,game-loop
.
- FLAGS
-
--additional-apks
=APK
,[APK
,…]- A list of up to 100 additional APKs to install, in addition to those being directly tested. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
--app-package
=APP_PACKAGE
-
(REMOVED) The Java package of the application under test. By default, the
application package name is parsed from the APK manifest.
Flag --app-package has been removed.
--async
- Invoke a test asynchronously without waiting for test results.
--auto-google-login
- Automatically log into the test device using a preconfigured Google account before beginning the test. Enabled by default, use --no-auto-google-login to disable.
--client-details
=[KEY
=VALUE
,…]-
Comma-separated, KEY=VALUE map of additional details to attach to the test
matrix. Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide
additional context about the tests being run. When consuming the test results,
such as in Cloud Functions or a CI system, these details can add additional
context such as a link to the corresponding pull request.
Example:
--client-details=buildNumber=1234,pullRequest=https://example.com/link/to/pull-request
To help you identify and locate your test matrix in the Firebase console, use the matrixLabel key.
Example:
--client-details=matrixLabel="Example matrix label"
--directories-to-pull
=[DIR_TO_PULL
,…]-
A list of paths that will be copied from the device's storage to the designated
results bucket after the test is complete. These must be absolute paths under
/sdcard
,/storage
, or/data/local/tmp
(for example,--directories-to-pull /sdcard/tempDir1,/data/local/tmp/tempDir2
). Path names are restricted to the charactersa-zA-Z0-9_-./+
. The paths/sdcard
and/data
will be made available and treated as implicit path substitutions. E.g. if/sdcard
on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device. Note that access to some directories on API levels 29 and later may also be limited by scoped storage rules. --environment-variables
=[KEY
=VALUE
,…]-
A comma-separated, key=value map of environment variables and their desired
values. The environment variables are mirrored as extra options to the
am instrument -e KEY1 VALUE1 …
command and passed to your test runner (typically AndroidJUnitRunner). Examples:Enable code coverage and provide a directory to store the coverage results when using Android Test Orchestrator (
--use-orchestrator
):--environment-variables clearPackageData=true,coverage=true,coverageFilePath=/sdcard/Download/
Enable code coverage and provide a file path to store the coverage results when
not
using Android Test Orchestrator (--no-use-orchestrator
):--environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec
Note: If you need to embed a comma into a
VALUE
string, please refer togcloud topic escaping
for ways to change the default list delimiter. --network-profile
=PROFILE_ID
-
The name of the network traffic profile, for example
--network-profile=LTE
, which consists of a set of parameters to emulate network conditions when running the test (default: no network shaping; see available profiles listed by the $ gcloud firebase test network-profiles list command). This feature only works on physical devices. --num-flaky-test-attempts
=int
-
Specifies the number of times a test execution should be reattempted if one or
more of its test cases fail for any reason. An execution that initially fails
but succeeds on any reattempt is reported as FLAKY.
The maximum number of reruns allowed is 10. (Default: 0, which implies no reruns.) All additional attempts are executed in parallel.
--obb-files
=OBB_FILE
,[OBB_FILE
]- A list of one or two Android OBB file names which will be copied to each test device before the tests will run (default: None). Each OBB file name must conform to the format as specified by Android (e.g. [main|patch].0300110.com.example.android.obb) and will be installed into <shared-storage>/Android/obb/<package-name>/ on the test device.
--other-files
=DEVICE_PATH
=FILE_PATH
,[…]-
A list of device-path=file-path pairs that indicate the device paths to push
files to the device before starting tests, and the paths of files to push.
Device paths must be under absolute, approved paths (${EXTERNAL_STORAGE}, or ${ANDROID_DATA}/local/tmp). Source file paths may be in the local filesystem or in Google Cloud Storage (gs://…).
Examples:
--other-files /sdcard/dir1/file1.txt=local/file.txt,/storage/dir2/file2.jpg=gs://bucket/file.jpg
This flag only copies files to the device. To install files, like OBB or APK files, see --obb-files and --additional-apks.
--performance-metrics
- Monitor and record performance metrics: CPU, memory, network usage, and FPS (game-loop only). Enabled by default, use --no-performance-metrics to disable.
--record-video
- Enable video recording during the test. Enabled by default, use --no-record-video to disable.
--results-bucket
=RESULTS_BUCKET
- The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
--results-dir
=RESULTS_DIR
-
The name of a
unique
Google Cloud Storage object within the results bucket where raw test results will be stored (default: a timestamp with a random suffix). Caution: if specified, this argumentmust be unique
for each test matrix you create, otherwise results from multiple test matrices will be overwritten or intermingled. --results-history-name
=RESULTS_HISTORY_NAME
- The history name for your test results (an arbitrary string label; default: the application's label from the APK manifest). All tests which use the same history name will have their results grouped together in the Firebase console in a time-ordered test history list.
- ANDROID GAME-LOOP TEST FLAGS
-
--scenario-labels
=LABEL
,[LABEL
,…]-
A list of game-loop scenario labels (default: None). Each game-loop scenario may
be labeled in the APK manifest file with one or more arbitrary strings, creating
logical groupings (e.g. GPU_COMPATIBILITY_TESTS). If
--scenario-numbers
and--scenario-labels
are specified together, Firebase Test Lab will first execute each scenario from--scenario-numbers
. It will then expand each given scenario label into a list of scenario numbers marked with that label, and execute those scenarios. --scenario-numbers
=int
,[int
,…]-
A list of game-loop scenario numbers which will be run as part of the test
(default: all scenarios). A maximum of 1024 scenarios may be specified in one
test matrix, but the maximum number may also be limited by the overall test
--timeout
setting.
- ANDROID INSTRUMENTATION TEST FLAGS
-
--test-package
=TEST_PACKAGE
-
(REMOVED) The Java package name of the instrumentation test. By default, the
test package name is parsed from the APK manifest.
Flag --test-package has been removed.
--test-runner-class
=TEST_RUNNER_CLASS
- The fully-qualified Java class name of the instrumentation test runner (default: the last name extracted from the APK manifest).
--test-targets
=TEST_TARGET
,[TEST_TARGET
,…]-
A list of one or more test target filters to apply (default: run all test
targets). Each target filter must be fully qualified with the package name,
class name, or test annotation desired. Any test filter supported by
am instrument -e …
is supported. See https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner for more information. Examples:-
--test-targets "package com.my.package.name"
-
--test-targets "notPackage com.package.to.skip"
-
--test-targets "class com.foo.ClassName"
-
--test-targets "notClass com.foo.ClassName#testMethodToSkip"
-
--test-targets "annotation com.foo.AnnotationToRun"
-
--test-targets "size large notAnnotation com.foo.AnnotationToSkip"
-
--use-orchestrator
- Whether each test runs in its own Instrumentation instance with the Android Test Orchestrator (default: Orchestrator is not used, same as specifying --no-use-orchestrator). Orchestrator is only compatible with AndroidJUnitRunner v1.1 or higher. See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator for more information about Android Test Orchestrator.
- ANDROID ROBO TEST FLAGS
-
--resign
-
Make Robo re-sign the app-under-test APK for a higher quality crawl. If your app
cannot properly function when re-signed, disable this feature. When an
app-under-test APK is not re-signed, Robo crawl is slower and Robo has access to
less information about the state of the crawled app, which reduces crawl
quality. Consequently, if your Roboscript has actions on elements of
RecyclerView or AdapterView, and you disable APK re-signing, those actions might
require manual tweaking because Robo does not identify RecyclerView and
AdapterView in this mode. Enabled by default, use
--no-resign
to disable. --robo-directives
=[TYPE
:RESOURCE_NAME
=INPUT
,…]-
A comma-separated (
<type>:<key>=<value>
) map ofrobo_directives
that you can use to customize the behavior of Robo test. Thetype
specifies the action type of the directive, which may take on valuesclick
,text
orignore
. If notype
is provided,text
will be used by default. Each key should be the Android resource name of a target UI element and each value should be the text input for that element. Values are only permitted fortext
type elements, so no value should be specified forclick
andignore
type elements. No more than oneclick
element is allowed.To provide custom login credentials for your app, use
--robo-directives text:username_resource=username,text:password_resource=password
To instruct Robo to click on the sign-in button, use
--robo-directives click:sign_in_button=
To instruct Robo to ignore any UI elements with resource names which equal or start with the user-defined value, use
--robo-directives ignore:ignored_ui_element_resource_name=
To learn more about Robo test and robo_directives, see https://firebase.google.com/docs/test-lab/android/command-line#custom_login_and_text_input_with_robo_test.
Caution: You should only use credentials for test accounts that are not associated with real users.
--robo-script
=ROBO_SCRIPT
- The path to a Robo Script JSON file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. You can guide the Robo test to perform specific actions by recording a Robo Script in Android Studio and then specifying this argument. Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting.
- DEPRECATED DEVICE DIMENSIONS FLAGS
-
--device-ids
=MODEL_ID
,[MODEL_ID
,…],-d
MODEL_ID
,[MODEL_ID
,…]-
The list of MODEL_IDs to test against (default: one device model determined by
the Firebase Test Lab device catalog; see TAGS listed by the
$ gcloud firebase test android models list
command). --locales
=LOCALE
,[LOCALE
,…],-l
LOCALE
,[LOCALE
,…]- The list of LOCALEs to test against (default: a single locale determined by the Firebase Test Lab device catalog).
--orientations
=ORIENTATION
,[ORIENTATION
],-o
ORIENTATION
,[ORIENTATION
]-
The device orientation(s) to test against (default: portrait). Specifying
'default' will pick the preferred orientation for the app.
ORIENTATION
must be one of:portrait
,landscape
,default
. --os-version-ids
=OS_VERSION_ID
,[…],-v
OS_VERSION_ID
,[…]- The list of OS_VERSION_IDs to test against (default: a version ID determined by the Firebase Test Lab device catalog).
- LIST COMMAND FLAGS
-
--filter
=EXPRESSION
-
Apply a Boolean filter
EXPRESSION
to each resource item to be listed. If the expression evaluatesTrue
, then that item is listed. For more details and examples of filter expressions, run $ gcloud topic filters. This flag interacts with other flags that are applied in this order:--flatten
,--sort-by
,--filter
,--limit
. --limit
=LIMIT
-
Maximum number of resources to list. The default is
unlimited
. This flag interacts with other flags that are applied in this order:--flatten
,--sort-by
,--filter
,--limit
. --page-size
=PAGE_SIZE
-
Some services group resource list output into pages. This flag specifies the
maximum number of resources per page. The default is determined by the service
if it supports paging, otherwise it is
unlimited
(no paging). Paging may be applied before or after--filter
and--limit
depending on the service. --sort-by
=[FIELD
,…]-
Comma-separated list of resource field key names to sort by. The default order
is ascending. Prefix a field with ``~´´ for descending order on that
field. This flag interacts with other flags that are applied in this order:
--flatten
,--sort-by
,--filter
,--limit
.
- GCLOUD WIDE FLAGS
-
These flags are available to all commands:
--access-token-file
,--account
,--billing-project
,--configuration
,--flags-file
,--flatten
,--format
,--help
,--impersonate-service-account
,--log-http
,--project
,--quiet
,--trace-token
,--user-output-enabled
,--verbosity
.Run
$ gcloud help
for details. - NOTES
-
These variants are also available:
gcloud alpha firebase test android run
gcloud beta firebase test android run
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-04-30 UTC.
[{
"type": "thumb-down",
"id": "hardToUnderstand",
"label":"Hard to understand"
},{
"type": "thumb-down",
"id": "incorrectInformationOrSampleCode",
"label":"Incorrect information or sample code"
},{
"type": "thumb-down",
"id": "missingTheInformationSamplesINeed",
"label":"Missing the information/samples I need"
},{
"type": "thumb-down",
"id": "otherDown",
"label":"Other"
}]
[{
"type": "thumb-up",
"id": "easyToUnderstand",
"label":"Easy to understand"
},{
"type": "thumb-up",
"id": "solvedMyProblem",
"label":"Solved my problem"
},{
"type": "thumb-up",
"id": "otherUp",
"label":"Other"
}]
{
"last_modified": "Last updated 2024-04-30 UTC.",
"state": ""
}