Secret Manager credential provider

Overview

The Secret Manager can safeguard your sensitive data, such as your API keys, passwords, and certificates. Use it to manage, access, and audit your secrets across Google Cloud.

The GoogleHadoopSecretManagerCredentialProvider API, which integrates with the Secret Manager, is an implementation of the Hadoop CredentialProvider API, a solution for securing sensitive credentials against public access.

You can use the Hadoop credential provider API in the following ways:

Terminology

The following table describes terms used in this document.

Term Description
Secret A secret is a project-global object that contains a collection of metadata and secret versions. Secret Manager lets you store, manage, and access secrets as binary blobs or text strings.
Credential In Hadoop and other Dataproc hosted applications, a credential consists of a credential name (ID) and credential value (password). A credential ID and value map to a secret ID and secret value (secret version) in Secret Manager.

Hadoop credential commands

You can use hadoop credential command to create, list, and manage secrets. hadoop credential commands use this general format: hadoop credential SUBCOMMAND OPTIONS.

In the following examples, the -provider flag is added to specify the provider type and location (the provider store). The gsm:// scheme specifies the Secret Manager.

  • Create a secret with the specified secret ID. The command doesn't create the secret if the specified secret ID exists. This behavior is consistent with the Hadoop CredentialProvider API. 

    hadoop credential create secret-id -provider gsm://projects/PROJECT_ID -v VALUE

  • List secrets stored in a project. 

    hadoop credential list -provider gsm://projects/PROJECT_ID

  • Check if a secret exists in a project with a specified value. 

    hadoop credential check SECRET_ID -provider gsm://projects/PROJECT_ID -v VALUE

  • Check for a specific secret version in a config file.

    hadoop credential conf CONFIG_FILE check SECRET_ID -provider gsm://projects/project-id -v VALUE

  • CONFIG_FILE: THe XML file that sets hadoop.security.credstore.google-secret-manager.secret-version.

  • Delete all versions of a secret in a project. 

    hadoop credential delete SECRET_ID -provider gsm://projects/ PROJECT_ID

See the Hadoop Commands Guide for more information.

Configure OSS components

You can configure Hadoop and other supported other OSS components to work with the Secret Manager by setting the following component properties:

  • Provider path (required): The provider path property, hadoop.security.credential.provider.path, is a comma-separated list of one or more credential provider URIs that is traversed to resolve a credential.

    --properties=hadoop.security.credential.provider.path=gsm://projects/project-id
    • A scheme is used to indicate the type of credential provider. Hadoop schemes include jceks://, user://,localjceks://. Use the gsm:// scheme as shown in the previous example to search for the credential in Secret Manager.

  • Substitute dot operator (optional): The Secret Manager doesn't support the dot(.) operator in secret names, but OSS component credential keys can contain this operator. When this property is set to true. you can replace dot(.)s with hyphen(-)s in credential names. For example, when this property is set to true, you can specify the credential name a.b.c as a-b-c when passing it to Secret Manager. This property is only needed to specify a credential using Hadoop credential commands or when an OSS component tries to resolve credentials. It has no effect on create, list, or delete Hadoop credential commands.

    --properties=hadoop.security.credstore.google-secret-manager.secret-id.substitute-dot-operator=true

  • Secret version (optional): Secrets in Secret Manager can have multiple versions (values). Use this property to access a secret version. By default, the Secret Manager accesses the LATEST version, which resolves to the latest value of the secret at runtime. A best practice is to define this property for stable access in production environments.

    --properties=hadoop.security.credstore.google-secret-manager.secret-version=1

Hive Metastore examples

The Hive Metastore property, javax.jdo.option.ConnectionPassword, contains the password used to authenticate access to a metastore database. This password is saved in plain-text format in hive-site.xml, which represents security risk. A production best-practice is to store the password in Secret Manager, then update the hive-site.xml config file to allow the Hive Metastore service to read the password from Secret Manager.

The following sections provide instructions on how to use Secret Manager in different Hive Metastore scenarios.

Hive cluster with local metastore

  1. Run the following command locally or in Cloud Shell to create a Dataproc cluster with the required cluster properties. 

    gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    ...other flags as needed... \
    --properties="hive:hadoop.security.credential.provider.path=gsm://projects/PROJECT_ID,hive:hadoop.security.credstore.google-secret-manager.secret-id.substitute-dot-operator=true"

  2. Create a secret. You can create a secret using the Secret Manager or the hadoop credential command.

    • Alternative 1: Use the Secret Manager to create a secret

      • Secret name: /projects/PROJECT_ID/secrets/javax-jdo-option-ConnectionPassword/versions/1
      • Secret value: METASTORE_PASSWORD.

    • Alternative 2: Use the hadoop credential command to create a secret. 

      sudo hadoop credential create javax-jdo-option-ConnectionPassword -provider gsm://projects/PROJECT_ID -v METASTORE_PASSWORD

      • METASTORE_PASSWORD: Since the Secret Manager does not support the dot(.) operator, substitute any dot(.)s in the password with hyphen(-)s.

  3. Verify that the secret exists. 

    sudo hadoop credential list -provider gsm://projects/PROJECT_ID

  4. Remove javax.jdo.option.ConnectionPassword from the hive-site.xml file. The following command opens the file for editing in vim. 

    sudo vim /etc/hive/conf/hive-site.xml

  5. Restart Hive Metastore. 

    sudo systemctl restart hive-metastore

Hive cluster with external metastore

  1. Run the following command locally or in Cloud Shell to create a Dataproc cluster with the following cluster properties. 

    gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    ...other flags as needed... \
    --properties=core:fs.defaultFS=gs://METASTORE_CLUSTER_PROXY_BUCKET,dataproc:dataproc.components.deactivate="hdfs hive-server2 hive-metastore"

  2. Create a secret. You can create a secret using the Secret Manager or the hadoop credential command.

    • Alternative 1: Use the Secret Manager to create a secret:
      • Secret name: /projects/PROJECT_ID/secrets/javax-jdo-option-ConnectionPassword/versions/1
      • Secret value: METASTORE_PASSWORD.
    • Alternative 2: Use the hadoop credential command to create a secret. 
      sudo hadoop credential create javax-jdo-option-ConnectionPassword -provider gsm://projects/PROJECT_ID -v METASTORE_PASSWORD
      • METASTORE_PASSWORD: Since the Secret Manager does not support the dot(.) operator, substitute dot(.)s in the password with hyphen(-)s.

  3. Verify that the secret exists. 

    sudo hadoop credential list -provider gsm://projects/PROJECT_ID

  4. Run the following command locally or in Cloud Shell to create a Dataproc cluster with the following cluster properties. Use this cluster to run Hive jobs and connects to the external metastore. 

    gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    ...other flags as needed...
    --properties="hive:javax.jdo.option.ConnectionURL=jdbc:mysql://metastore-cluster-name-m/metastore,hive:hadoop.security.credential.provider.path=gsm://projects/project-id,hive:hadoop.security.credstore.google-secret-manager.secret-id.substitute-dot-operator=true"

For more information