You can choose the database type for your Dataproc Metastore service to fit your performance needs. You can choose between a MySQL or Spanner database type.
MySQL is the default database type. It's supported for all Hive versions. Spanner-backed Dataproc Metastore services are supported for Hive versions 2.3.6 and 3.1.2.
Differences between MySQL and Spanner database types
The MySQL database type uses Cloud SQL, while the Spanner database type uses Cloud Spanner. MySQL supports encryption features, but lacks scalability. Spanner provides more reliability with the trade-off of increased latency.
The following table lists the differences between MySQL and Spanner database types:
MySQL | Spanner | |
---|---|---|
Reliability (uptime) | Cloud SQL SLO 99.95%* | Cloud Spanner SLO 99.99%* |
Latency | No change in latency | Increased latency of about 50% |
CMEK | Supported | Not Supported |
Maintenance windows | Required | Not Required |
Notes:
- *The Cloud SQL and Cloud Spanner SLOs don't directly translate to Dataproc Metastore SLOs. Dataproc Metastore SLOs are not affected by your database type selection.
- There is no pricing difference between the two database types.
Choose the database type for a new Dataproc Metastore service
You can choose the database type for your Dataproc Metastore service when you create it.
Console
In the console, open the Dataproc Metastore page:
At the top of the Dataproc Metastore page, click the Create button. The Create service page opens.
Configure your service as desired.
Under Database type, select MySQL or Spanner. MySQL is the default database type.
Click Submit.
gcloud
Run the following
gcloud beta metastore services create
command:gcloud beta metastore services create SERVICE \ --location=LOCATION \ --database-type=DATABASE_TYPE; default="mysql"
Verify that the creation was successful.
Database type caveats
Spanner-backed Dataproc Metastore services have the following caveats:
Only support Hive versions 2.3.6 and 3.1.2.
Don't support Dataproc Metastore service to Data Catalog sync.
Not supported with Auxiliary versions.
Don't support import, export, backup, and restore.