- Resource: Service
- JSON representation
- HiveMetastoreConfig
- KerberosConfig
- Secret
- EndpointProtocol
- AuxiliaryVersionConfig
- NetworkConfig
- Consumer
- State
- Tier
- MetadataIntegration
- DataCatalogConfig
- DataplexConfig
- Lake
- MaintenanceWindow
- MetadataManagementActivity
- MetadataExport
- State
- Restore
- State
- RestoreType
- ReleaseChannel
- EncryptionConfig
- DatabaseType
- TelemetryConfig
- LogFormat
- ScalingConfig
- InstanceSize
- AutoscalingConfig
- LimitConfig
- MultiRegionConfig
- RootCACertificate
- CustomRegionConfig
- ScheduledBackup
- LatestBackup
- State
- Methods
Resource: Service
A managed metastore service that serves metadata queries.
JSON representation |
---|
{ "name": string, "createTime": string, "updateTime": string, "labels": { string: string, ... }, "network": string, "endpointUri": string, "port": integer, "state": enum ( |
Fields | |
---|---|
name |
Immutable. The relative resource name of the metastore service, in the following format:
|
createTime |
Output only. The time when the metastore service was created. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
updateTime |
Output only. The time when the metastore service was last updated. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
labels |
User-defined labels for the metastore service. An object containing a list of |
network |
Immutable. The relative resource name of the VPC network on which the instance can be accessed. It is specified in the following form:
|
endpointUri |
Output only. The URI of the endpoint used to access the metastore service. |
port |
The TCP port at which the metastore service is reached. Default: 9083. |
state |
Output only. The current state of the metastore service. |
stateMessage |
Output only. Additional information about the current state of the metastore service, if available. |
artifactGcsUri |
Output only. A Cloud Storage URI (starting with |
tier |
The tier of the service. |
metadataIntegration |
Optional. The setting that defines how metastore metadata should be integrated with external services and systems. |
maintenanceWindow |
The one hour maintenance window of the metastore service. This specifies when the service can be restarted for maintenance purposes in UTC time. Maintenance window is not needed for services with the SPANNER database type. |
uid |
Output only. The globally unique resource identifier of the metastore service. |
metadataManagementActivity |
Output only. The metadata management activities of the metastore service. |
releaseChannel |
Immutable. The release channel of the service. If unspecified, defaults to |
encryptionConfig |
Immutable. Information used to configure the Dataproc Metastore service to encrypt customer data at rest. Cannot be updated. |
networkConfig |
The configuration specifying the network settings for the Dataproc Metastore service. |
databaseType |
Immutable. The database type that the Metastore service stores its data. |
telemetryConfig |
The configuration specifying telemetry settings for the Dataproc Metastore service. If unspecified defaults to |
scalingConfig |
Scaling configuration of the metastore service. |
multiRegionConfig |
Optional. Specifies the multi-region configuration information for the Hive metastore service. |
scheduledBackup |
Optional. The configuration of scheduled backup for the metastore service. |
deletionProtection |
Optional. Indicates if the dataproc metastore should be protected against accidental deletions. |
Union field metastore_config . Configuration properties specific to the underlying metastore service technology (the software that serves metastore queries). metastore_config can be only one of the following: |
|
hiveMetastoreConfig |
Configuration information specific to running Hive metastore software as the metastore service. |
HiveMetastoreConfig
Specifies configuration information specific to running Hive metastore software as the metastore service.
JSON representation |
---|
{ "version": string, "configOverrides": { string: string, ... }, "kerberosConfig": { object ( |
Fields | |
---|---|
version |
Immutable. The Hive metastore schema version. |
configOverrides |
A mapping of Hive metastore configuration key-value pairs to apply to the Hive metastore (configured in An object containing a list of |
kerberosConfig |
Information used to configure the Hive metastore service as a service principal in a Kerberos realm. To disable Kerberos, use the |
endpointProtocol |
The protocol to use for the metastore service endpoint. If unspecified, defaults to |
auxiliaryVersions |
A mapping of Hive metastore version to the auxiliary version configuration. When specified, a secondary Hive metastore service is created along with the primary service. All auxiliary versions must be less than the service's primary version. The key is the auxiliary service name and it must match the regular expression a-z?. This means that the first character must be a lowercase letter, and all the following characters must be hyphens, lowercase letters, or digits, except the last character, which cannot be a hyphen. An object containing a list of |
KerberosConfig
Configuration information for a Kerberos principal.
JSON representation |
---|
{
"keytab": {
object ( |
Fields | |
---|---|
keytab |
A Kerberos keytab file that can be used to authenticate a service principal with a Kerberos Key Distribution Center (KDC). |
principal |
A Kerberos principal that exists in the both the keytab the KDC to authenticate as. A typical principal is of the form |
krb5ConfigGcsUri |
A Cloud Storage URI that specifies the path to a krb5.conf file. It is of the form |
Secret
A securely stored value.
JSON representation |
---|
{ // Union field |
Fields | |
---|---|
Union field
|
|
cloudSecret |
The relative resource name of a Secret Manager secret version, in the following form:
|
EndpointProtocol
Protocols available for serving the metastore service endpoint.
Enums | |
---|---|
ENDPOINT_PROTOCOL_UNSPECIFIED |
The protocol is not set. |
THRIFT |
Use the legacy Apache Thrift protocol for the metastore service endpoint. |
GRPC |
Use the modernized gRPC protocol for the metastore service endpoint. |
AuxiliaryVersionConfig
Configuration information for the auxiliary service versions.
JSON representation |
---|
{
"version": string,
"configOverrides": {
string: string,
...
},
"networkConfig": {
object ( |
Fields | |
---|---|
version |
The Hive metastore version of the auxiliary service. It must be less than the primary Hive metastore service's version. |
configOverrides |
A mapping of Hive metastore configuration key-value pairs to apply to the auxiliary Hive metastore (configured in An object containing a list of |
networkConfig |
Output only. The network configuration contains the endpoint URI(s) of the auxiliary Hive metastore service. |
NetworkConfig
Network configuration for the Dataproc Metastore service.
JSON representation |
---|
{
"consumers": [
{
object ( |
Fields | |
---|---|
consumers[] |
Immutable. The consumer-side network configuration for the Dataproc Metastore instance. |
customRoutesEnabled |
Enables custom routes to be imported and exported for the Dataproc Metastore service's peered VPC network. |
Consumer
Contains information of the customer's network configurations.
JSON representation |
---|
{ "endpointUri": string, "endpointLocation": string, // Union field |
Fields | |
---|---|
endpointUri |
Output only. The URI of the endpoint used to access the metastore service. |
endpointLocation |
Output only. The location of the endpoint URI. Format: |
Union field
|
|
subnetwork |
Immutable. The subnetwork of the customer project from which an IP address is reserved and used as the Dataproc Metastore service's endpoint. It is accessible to hosts in the subnet and to all hosts in a subnet in the same region and same network. There must be at least one IP address available in the subnet's primary range. The subnet is specified in the following form:
|
State
The current state of the metastore service.
Enums | |
---|---|
STATE_UNSPECIFIED |
The state of the metastore service is unknown. |
CREATING |
The metastore service is in the process of being created. |
ACTIVE |
The metastore service is running and ready to serve queries. |
SUSPENDING |
The metastore service is entering suspension. Its query-serving availability may cease unexpectedly. |
SUSPENDED |
The metastore service is suspended and unable to serve queries. |
UPDATING |
The metastore service is being updated. It remains usable but cannot accept additional update requests or be deleted at this time. |
DELETING |
The metastore service is undergoing deletion. It cannot be used. |
ERROR |
The metastore service has encountered an error and cannot be used. The metastore service should be deleted. |
AUTOSCALING |
The Dataproc Metastore service 2 is being scaled up or down. |
MIGRATING |
The metastore service is processing a managed migration. |
Tier
Available service tiers.
Enums | |
---|---|
TIER_UNSPECIFIED |
The tier is not set. |
DEVELOPER |
The developer tier provides limited scalability and no fault tolerance. Good for low-cost proof-of-concept. |
ENTERPRISE |
The enterprise tier provides multi-zone high availability, and sufficient scalability for enterprise-level Dataproc Metastore workloads. |
MetadataIntegration
Specifies how metastore metadata should be integrated with external services.
JSON representation |
---|
{ "dataCatalogConfig": { object ( |
Fields | |
---|---|
dataCatalogConfig |
Optional. The integration config for the Data Catalog service. |
dataplexConfig |
The integration config for the Dataplex service. |
DataCatalogConfig
Specifies how metastore metadata should be integrated with the Data Catalog service.
JSON representation |
---|
{ "enabled": boolean } |
Fields | |
---|---|
enabled |
Optional. Defines whether the metastore metadata should be synced to Data Catalog. The default value is to disable syncing metastore metadata to Data Catalog. |
DataplexConfig
Specifies how metastore metadata should be integrated with the Dataplex service.
JSON representation |
---|
{
"lakeResources": {
string: {
object ( |
Fields | |
---|---|
lakeResources |
A reference to the Lake resources that this metastore service is attached to. The key is the lake resource name. Example: An object containing a list of |
Lake
Represents a Lake resource
JSON representation |
---|
{ "name": string } |
Fields | |
---|---|
name |
The Lake resource name. Example: |
MaintenanceWindow
Maintenance window. This specifies when Dataproc Metastore may perform system maintenance operation to the service.
JSON representation |
---|
{
"hourOfDay": integer,
"dayOfWeek": enum ( |
Fields | |
---|---|
hourOfDay |
The hour of day (0-23) when the window starts. |
dayOfWeek |
The day of week, when the window starts. |
MetadataManagementActivity
The metadata management activities of the metastore service.
JSON representation |
---|
{ "metadataExports": [ { object ( |
Fields | |
---|---|
metadataExports[] |
Output only. The latest metadata exports of the metastore service. |
restores[] |
Output only. The latest restores of the metastore service. |
MetadataExport
The details of a metadata export operation.
JSON representation |
---|
{ "startTime": string, "endTime": string, "state": enum ( |
Fields | |
---|---|
startTime |
Output only. The time when the export started. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
endTime |
Output only. The time when the export ended. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
state |
Output only. The current state of the export. |
databaseDumpType |
Output only. The type of the database dump. |
Union field
|
|
destinationGcsUri |
Output only. A Cloud Storage URI of a folder that metadata are exported to, in the form of |
State
The current state of the metadata export.
Enums | |
---|---|
STATE_UNSPECIFIED |
The state of the metadata export is unknown. |
RUNNING |
The metadata export is running. |
SUCCEEDED |
The metadata export completed successfully. |
FAILED |
The metadata export failed. |
CANCELLED |
The metadata export is cancelled. |
Restore
The details of a metadata restore operation.
JSON representation |
---|
{ "startTime": string, "endTime": string, "state": enum ( |
Fields | |
---|---|
startTime |
Output only. The time when the restore started. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
endTime |
Output only. The time when the restore ended. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
state |
Output only. The current state of the restore. |
backup |
Output only. The relative resource name of the metastore service backup to restore from, in the following form:
|
type |
Output only. The type of restore. |
details |
Output only. The restore details containing the revision of the service to be restored to, in format of JSON. |
backupLocation |
Optional. A Cloud Storage URI specifying where the backup artifacts are stored, in the format |
State
The current state of the restore.
Enums | |
---|---|
STATE_UNSPECIFIED |
The state of the metadata restore is unknown. |
RUNNING |
The metadata restore is running. |
SUCCEEDED |
The metadata restore completed successfully. |
FAILED |
The metadata restore failed. |
CANCELLED |
The metadata restore is cancelled. |
RestoreType
The type of restore. If unspecified, defaults to METADATA_ONLY
.
Enums | |
---|---|
RESTORE_TYPE_UNSPECIFIED |
The restore type is unknown. |
FULL |
The service's metadata and configuration are restored. |
METADATA_ONLY |
Only the service's metadata is restored. |
ReleaseChannel
Release channels bundle features of varying levels of stability. Newer features may be introduced initially into less stable release channels and can be automatically promoted into more stable release channels.
Enums | |
---|---|
RELEASE_CHANNEL_UNSPECIFIED |
Release channel is not specified. |
CANARY |
The CANARY release channel contains the newest features, which may be unstable and subject to unresolved issues with no known workarounds. Services using the CANARY release channel are not subject to any SLAs. |
STABLE |
The STABLE release channel contains features that are considered stable and have been validated for production use. |
EncryptionConfig
Encryption settings for the service.
JSON representation |
---|
{ "kmsKey": string } |
Fields | |
---|---|
kmsKey |
The fully qualified customer provided Cloud KMS key name to use for customer data encryption, in the following format:
|
DatabaseType
The backend database type for the metastore service.
Enums | |
---|---|
DATABASE_TYPE_UNSPECIFIED |
The DATABASE_TYPE is not set. |
MYSQL |
MySQL is used to persist the metastore data. |
SPANNER |
Spanner is used to persist the metastore data. |
TelemetryConfig
Telemetry Configuration for the Dataproc Metastore service.
JSON representation |
---|
{
"logFormat": enum ( |
Fields | |
---|---|
logFormat |
The output format of the Dataproc Metastore service's logs. |
LogFormat
Enums | |
---|---|
LOG_FORMAT_UNSPECIFIED |
The LOG_FORMAT is not set. |
LEGACY |
Logging output uses the legacy textPayload format. |
JSON |
Logging output uses the jsonPayload format. |
ScalingConfig
Represents the scaling configuration of a metastore service.
JSON representation |
---|
{ "autoscalingConfig": { object ( |
Fields | |
---|---|
autoscalingConfig |
Optional. The autoscaling configuration. |
Union field scaling_model . Represents either a predetermined instance size or a numeric scaling factor. scaling_model can be only one of the following: |
|
instanceSize |
An enum of readable instance sizes, with each instance size mapping to a float value (e.g. InstanceSize.EXTRA_SMALL = scalingFactor(0.1)) |
scalingFactor |
Scaling factor, increments of 0.1 for values less than 1.0, and increments of 1.0 for values greater than 1.0. |
InstanceSize
Metastore instance sizes.
Enums | |
---|---|
INSTANCE_SIZE_UNSPECIFIED |
Unspecified instance size |
EXTRA_SMALL |
Extra small instance size, maps to a scaling factor of 0.1. |
SMALL |
Small instance size, maps to a scaling factor of 0.5. |
MEDIUM |
Medium instance size, maps to a scaling factor of 1.0. |
LARGE |
Large instance size, maps to a scaling factor of 3.0. |
EXTRA_LARGE |
Extra large instance size, maps to a scaling factor of 6.0. |
AutoscalingConfig
Represents the autoscaling configuration of a metastore service.
JSON representation |
---|
{
"autoscalingFactor": number,
"autoscalingEnabled": boolean,
"limitConfig": {
object ( |
Fields | |
---|---|
autoscalingFactor |
Output only. The scaling factor of a service with autoscaling enabled. |
autoscalingEnabled |
Optional. Whether or not autoscaling is enabled for this service. |
limitConfig |
Optional. The LimitConfig of the service. |
LimitConfig
Represents the autoscaling limit configuration of a metastore service.
JSON representation |
---|
{ "maxScalingFactor": number, "minScalingFactor": number } |
Fields | |
---|---|
maxScalingFactor |
Optional. The highest scaling factor that the service should be autoscaled to. |
minScalingFactor |
Optional. The lowest scaling factor that the service should be autoscaled to. |
MultiRegionConfig
The multi-region config for the Dataproc Metastore service.
JSON representation |
---|
{ "certificates": [ { object ( |
Fields | |
---|---|
certificates[] |
Output only. The list of root CA certificates that a gRPC client uses to connect to a multi-regional Dataproc Metastore service. |
customRegionConfig |
|
RootCACertificate
A gRPC client must install all root CA certificates to connect to a multi-regional Dataproc Metastore service and achieve failover.
JSON representation |
---|
{ "certificate": string, "expirationTime": string } |
Fields | |
---|---|
certificate |
The root CA certificate in PEM format. The maximum length is 65536 bytes. |
expirationTime |
The certificate expiration time in timestamp format. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
CustomRegionConfig
Custom configuration used to specify regions that the metastore service runs in. Currently only supported in the us
multi-region.
JSON representation |
---|
{ "readWriteRegions": [ string ], "readOnlyRegions": [ string ] } |
Fields | |
---|---|
readWriteRegions[] |
Required. The list of read-write regions where the metastore service runs in. These regions should be part (or subset) of the multi-region. |
readOnlyRegions[] |
Optional. The list of read-only regions where the metastore service runs in. These regions should be part (or subset) of the multi-region. |
ScheduledBackup
This specifies the configuration of scheduled backup.
JSON representation |
---|
{
"enabled": boolean,
"cronSchedule": string,
"timeZone": string,
"nextScheduledTime": string,
"backupLocation": string,
"latestBackup": {
object ( |
Fields | |
---|---|
enabled |
Optional. Defines whether the scheduled backup is enabled. The default value is false. |
cronSchedule |
Optional. The scheduled interval in Cron format, see https://en.wikipedia.org/wiki/Cron The default is empty: scheduled backup is not enabled. Must be specified to enable scheduled backups. |
timeZone |
Optional. Specifies the time zone to be used when interpreting cronSchedule. Must be a time zone name from the time zone database (https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), e.g. America/Los_Angeles or Africa/Abidjan. If left unspecified, the default is UTC. |
nextScheduledTime |
Output only. The time when the next backups execution is scheduled to start. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
backupLocation |
Optional. A Cloud Storage URI of a folder, in the format |
latestBackup |
Output only. The details of the latest scheduled backup. |
LatestBackup
The details of the latest scheduled backup.
JSON representation |
---|
{
"backupId": string,
"startTime": string,
"state": enum ( |
Fields | |
---|---|
backupId |
Output only. The ID of an in-progress scheduled backup. Empty if no backup is in progress. |
startTime |
Output only. The time when the backup was started. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
state |
Output only. The current state of the backup. |
duration |
Output only. The duration of the backup completion. A duration in seconds with up to nine fractional digits, ending with ' |
State
The current state of the backup.
Enums | |
---|---|
STATE_UNSPECIFIED |
The state of the backup is unknown. |
IN_PROGRESS |
The backup is in progress. |
SUCCEEDED |
The backup completed. |
FAILED |
The backup failed. |
Methods |
|
---|---|
|
Alter metadata resource location. |
|
Alter metadata table properties. |
|
Cancels the ongoing Managed Migration process. |
|
Completes the managed migration process. |
|
Creates a metastore service in a project and location. |
|
Deletes a single service. |
|
Exports metadata from a service. |
|
Gets the details of a single service. |
|
Gets the access control policy for a resource. |
|
Lists services in a project and location. |
|
Move a table to another database. |
|
Updates the parameters of a single service. |
|
Query Dataproc Metastore metadata. |
|
Removes the attached IAM policies for a resource |
|
Restores a service from a backup. |
|
Sets the access control policy on the specified resource. |
|
Starts the Managed Migration process. |
|
Returns permissions that a caller has on the specified resource. |