gcloud beta metastore services create

NAME

gcloud beta metastore services create - create a Dataproc Metastore service

SYNOPSIS

gcloud beta metastore services create (SERVICE : --location=LOCATION) [--async] [--autoscaling-enabled] [--data-catalog-sync] [--database-type=DATABASE_TYPE; default="mysql"] [--deletion-protection] [--encryption-kms-key=ENCRYPTION_KMS_KEY] [--endpoint-protocol=ENDPOINT_PROTOCOL; default="thrift"] [--hive-metastore-version=HIVE_METASTORE_VERSION] [--labels=[KEY=VALUE,…]] [--port=PORT; default=9083] [--release-channel=RELEASE_CHANNEL; default="stable"] [--auxiliary-versions=[AUXILIARY_VERSIONS,…]     | --auxiliary-versions-from-file=AUXILIARY_VERSIONS_FROM_FILE] [--consumer-subnetworks=[CONSUMER_SUBNETWORKS,…]     | --network=NETWORK     | --network-config-from-file=NETWORK_CONFIG_FROM_FILE] [--hive-metastore-configs=[KEY=VALUE,…]     | --hive-metastore-configs-from-file=PATH_TO_FILE] [--instance-size=INSTANCE_SIZE     | --scaling-factor=SCALING_FACTOR     | --tier=TIER     | --max-scaling-factor=MAX_SCALING_FACTOR --min-scaling-factor=MIN_SCALING_FACTOR] [--kerberos-principal=KERBEROS_PRINCIPAL --keytab=KEYTAB --krb5-config=KRB5_CONFIG] [--maintenance-window-day=MAINTENANCE_WINDOW_DAY --maintenance-window-hour=MAINTENANCE_WINDOW_HOUR] [--read-only-regions=[READ_ONLY_REGIONS,…] --read-write-regions=[READ_WRITE_REGIONS,…]] [--scheduled-backup-configs-from-file=PATH_TO_FILE     | --enable-scheduled-backup --scheduled-backup-cron=SCHEDULED_BACKUP_CRON --scheduled-backup-location=SCHEDULED_BACKUP_LOCATION] [GCLOUD_WIDE_FLAG …]

DESCRIPTION

(BETA) Create a new Dataproc Metastore service with the given name and configurations.

If run asynchronously with --async, exits after printing one operation name that can be used to poll the status of the creation via:

gcloud metastore operations describe

EXAMPLES

To create a Dataproc Metastore service with the name my-metastore-service in location us-central using a non default port 9090, run:

gcloud beta metastore services create my-metastore-service --location=us-central1 --port=9090

To create a Dataproc Metastore service with the name my-metastore-service in location us-central using a non default network foo, run:

gcloud beta metastore services create my-metastore-service --location=us-central1 --network=foo

POSITIONAL ARGUMENTS

Service resource - Arguments and flags that specify the Dataproc Metastore service you want to create. The arguments in this group can be used to specify the attributes of this resource. (NOTE) Some attributes are not given arguments in this group but can be set in other ways.

To set the project attribute:

• provide the argument service on the command line with a fully specified name;
• provide the argument --project on the command line;
• set the property core/project.

This must be specified.

SERVICE

ID of the service or fully qualified identifier for the service.

To set the service attribute:

• provide the argument service on the command line.

This positional argument must be specified if any of the other arguments in this group are specified.

--location=LOCATION

The location of the Dataproc Metastore service.

If not specified, will use default metastore/location. To set the location attribute:

• provide the argument service on the command line with a fully specified name;
• provide the argument --location on the command line;
• set the property metastore/location.

FLAGS

--async

Return immediately, without waiting for the operation in progress to complete.

--autoscaling-enabled

A boolean flag to determine whether Dataproc Metastore autoscaling should be enabled, false if unspecified.

The default minimum and maximum scaling factors are 0.1 and 6.0, respectively.

The minimum and maximum scaling factors can be specified using --min-scaling-factor and --max-scaling-factor.

--data-catalog-sync

A boolean flag to determine whether Dataproc Metastore metadata sync to Data Catalog should be enabled, false if unspecified. Mutually exclusive with flag --encryption-kms-key.

--database-type=DATABASE_TYPE; default="mysql"

The type of database the Dataproc Metastore service will store data in. DATABASE_TYPE must be one of:

mysql

MYSQL database type is a Dataproc Metastore service backed by MySQL CloudSQL.

spanner

SPANNER database type is a Dataproc Metastore service backed by Cloud Spanner.

--deletion-protection

Flag that enables delete protection on Dataproc Metastore instance to prevent accidental deletions of the instance. Use --deletion-protection to enable.

--encryption-kms-key=ENCRYPTION_KMS_KEY

The name of the Cloud KMS key to use for customer data encryption. Cannot be changed once the service is created. Mutually exclusive with flag --data-catalog-sync. This can be provided as a fully-qualified URL, or a relative name in the form projects/{project_id}/locations/{location}/keyRings/{key_ring_id}/cryptoKeys/{crypto_key_id}.

--endpoint-protocol=ENDPOINT_PROTOCOL; default="thrift"

The protocol to use for the metastore service endpoint. If unspecified, defaults to THRIFT. ENDPOINT_PROTOCOL must be one of:

grpc

The modernized GRPC protocol.

thrift

The legacy Apache THRIFT protocol.

--hive-metastore-version=HIVE_METASTORE_VERSION

The Hive metastore schema version. The supported versions of a location are listed via:

gcloud metastore locations describe

If unspecified, the default version chosen by the server will be used.

--labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens (-), underscores (_), lowercase characters, and numbers. Values must contain only hyphens (-), underscores (_), lowercase characters, and numbers.

--port=PORT; default=9083

The TCP port on which the Metastore service will listen. If unspecified, the default port 9083 will be used.

--release-channel=RELEASE_CHANNEL; default="stable"

The release channel of the service. RELEASE_CHANNEL must be one of:

canary

The CANARY release channel contains the newest features, which may be unstable and subject to unresolved issues with no known workarounds. Services using the CANARY release channel are not subject to any SLAs.

stable

The STABLE release channel contains features that are considered stable and have been validated for production use.

Auxiliary versions configuration for the Dataproc Metastore service. When specified, a secondary Hive metastore service is created along with the primary service. At most one of these can be specified:

--auxiliary-versions=[AUXILIARY_VERSIONS,…]

Comma-separated list of auxiliary Hive metastore versions to deploy. Auxiliary Hive metastore versions must be less than the primary Hive metastore service's version.

--auxiliary-versions-from-file=AUXILIARY_VERSIONS_FROM_FILE

Path to a YAML file containing the auxiliary versions configuration for Dataproc Metastore instance. The file should contain a unique auxiliary service name and auxiliary version that is lower than the primary service version. The primary version's Hive metastore configs are applied to the auxiliary version. Additional Hive metastore configs can be specified for the auxiliary version using "config_overrides". If a specified config property has already been overridden in the primary Hive metastore version's configs, the auxiliary version's override takes precedence. The contents of the file should be structured as follows:

YAML:

  - name: aux-service1
    version: x.y.z
    config_overrides:
      key1: value1
      key2: value2
    …
  - name: aux-service2
    version: x.y.z
    config_overrides:
      key1: value1
      key2: value2
    …
  …

Network configuration for the metastore service. At most one of these can be specified:

--consumer-subnetworks=[CONSUMER_SUBNETWORKS,…]

The list of subnetworks from which the service can be accessed. This can be the subnetwork's ID, fully-qualified URL, or relative name in the form projects/{project_id}/regions/{region_id}/subnetworks/{subnetwork}.

--network=NETWORK

The name of the network on which the service can be accessed. This can be the network's ID, fully-qualified URL, or relative name in the form projects/{project_id}/global/networks/{network}. If "network", "consumer-subnetworks" and "network-config" are unspecified, the "default" network will be used.

--network-config-from-file=NETWORK_CONFIG_FROM_FILE

Path to a YAML file containing the network configuration for Dataproc Metastore instance. The contents of the file should be structured as follows:

YAML: consumers:

• subnetwork: projects/{project_id}/regions/{region_id}/subnetworks/{subnetwork_id}
• subnetwork: projects/{project_id}/regions/{region_id}/subnetworks/{subnetwork_id} …

Configuration properties specific to running Hive metastore software as the metastore service. At most one of these can be specified:

--hive-metastore-configs=[KEY=VALUE,…]

A mapping of Hive metastore configuration key-value pairs to apply to the Hive metastore.

--hive-metastore-configs-from-file=PATH_TO_FILE

Path to a XML file containing a mapping of Hive metastore configuration key-value pairs to apply to the Hive metastore.

For example:

hive-site.xml
    <configuration>
      <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>${test.warehouse.dir}</value>
        <description></description>
      </property>
    </configuration>

Use a full or relative path to a local file containing the value of hive_metastore_configs.

The instance size of the Dataproc Metastore instance. At most one of these can be specified:

--instance-size=INSTANCE_SIZE

The instance size of the service. INSTANCE_SIZE must be one of:

extra-large

Extra large instance size, maps to a scaling factor of 6.0.

extra-small

Extra small instance size, maps to a scaling factor of 0.1.

large

Large instance size, maps to a scaling factor of 3.0.

medium

Medium instance size, maps to a scaling factor of 1.0.

small

Small instance size, maps to a scaling factor of 0.5.

--scaling-factor=SCALING_FACTOR

The scaling factor of the service. Can be one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].

--tier=TIER

The tier of the service. TIER must be one of:

developer

The developer tier provides limited scalability and no fault tolerance. Good for low-cost proof-of-concept.

enterprise

The enterprise tier provides multi-zone high availability, and sufficient scalability for enterprise-level Dataproc Metastore workloads.

The autoscaling configuration of the Dataproc Metastore instance.

--max-scaling-factor=MAX_SCALING_FACTOR

The maximum scaling factor allowed by the autoscaler. Can be one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]. The default value is 6.0.

--min-scaling-factor=MIN_SCALING_FACTOR

The minimum scaling factor allowed by the autoscaler. Can be one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]. The default value is 0.1.

Information used to configure the Hive metastore service as a service principal in a Kerberos realm.

--kerberos-principal=KERBEROS_PRINCIPAL

A Kerberos principal that exists in the KDC to authenticate as. A typical principal is of the form "primary/instance@REALM", but there is no exact format.

This flag argument must be specified if any of the other arguments in this group are specified.

--keytab=KEYTAB

A Kerberos keytab file that can be used to authenticate a service principal with a Kerberos Key Distribution Center. This is a Secret Manager secret version, and can be fully-qualified URL, or relative name in the form projects/{project_id}/secrets/{secret_id}/versions/{version_id}.

This flag argument must be specified if any of the other arguments in this group are specified.

--krb5-config=KRB5_CONFIG

A Cloud Storage URI that specifies the path to a krb5.conf file. It is of the form gs://{bucket_name}/path/krb5.conf, although the file does not need to be named krb5.conf explicitly.

This flag argument must be specified if any of the other arguments in this group are specified.

The one hour maintenance window that specifies when Dataproc Metastore may perform system maintenance operation to the service, in UTC time.

--maintenance-window-day=MAINTENANCE_WINDOW_DAY

The day of week when the window starts, e.g., sun. MAINTENANCE_WINDOW_DAY must be one of: fri, mon, sat, sun, thu, tue, wed.

This flag argument must be specified if any of the other arguments in this group are specified.

--maintenance-window-hour=MAINTENANCE_WINDOW_HOUR

The hour of day (0-23) when the window starts.

This flag argument must be specified if any of the other arguments in this group are specified.

Custom region configuration for the metastore service.

--read-only-regions=[READ_ONLY_REGIONS,…]

The list of read-only regions where the metastore service runs in. These regions should be part (or subset) of the multi-region.

--read-write-regions=[READ_WRITE_REGIONS,…]

The list of read-write regions where the metastore service runs in. These regions should be part (or subset) of the multi-region.

Scheduled backup configurations to enable a scheduled backup of the metastore service. At most one of these can be specified:

--scheduled-backup-configs-from-file=PATH_TO_FILE

Path to a JSON file containing a mapping of scheduled backup configurations key-value pairs to apply to the scheduled backup.

For example:

backup_configs.json
    {
      "enabled": true,
      "cron_schedule": "0 * * * *",
      "backup_location": "gs://example-bucket",
      "time_zone": "UTC"
    }

Use a full or relative path to a local file containing the value of scheduled_backup_configs.

Scheduled backup configurations to enable a scheduled backup of the metastore service.

--enable-scheduled-backup

Enable or disable scheduled backups for the Metastore service.

--scheduled-backup-cron=SCHEDULED_BACKUP_CRON

Specify the frequency of a scheduled backup of the metastore service.

--scheduled-backup-location=SCHEDULED_BACKUP_LOCATION

Set the Cloud Storage URI to store the scheduled backups of the metastore service.

GCLOUD WIDE FLAGS

These flags are available to all commands: --access-token-file, --account, --billing-project, --configuration, --flags-file, --flatten, --format, --help, --impersonate-service-account, --log-http, --project, --quiet, --trace-token, --user-output-enabled, --verbosity.

Run $ gcloud help for details.

API REFERENCE

This command uses the metastore/v1beta API. The full documentation for this API can be found at: https://cloud.google.com/dataproc-metastore/docs

NOTES

This command is currently in beta and might change without notice. These variants are also available:

gcloud metastore services create

gcloud alpha metastore services create