Troubleshoot CMEK and default setting errors

This document describes how to find and mitigate common CMEK configuration errors, and it describes how to identify errors that occur when setting the default resource location.

Troubleshoot setting the default resource location

You attempt to update the default storage location for an organization or folder, but the command fails with an error similar to the following:

ERROR: (gcloud.logging.settings.update) INVALID_ARGUMENT: The KMS key location must match the storage location. Received KMS key location: us-central1, storage location: us-west1
- '@type': type.googleapis.com/google.rpc.DebugInfo
  detail: '[ORIGINAL ERROR] generic::invalid_argument: The KMS key location must match
    the storage location. Received KMS key location: us-central1, storage location:
    us-west1 [google.rpc.error_details_ext] { message: "The KMS key location must
    match the storage location. Received KMS key location: us-central1, storage location:
    us-west1" }'

To resolve this error, set the default storage location for the organization or folder to match the location of the Cloud Key Management Service key for the organization or folder.

Troubleshoot VPC Service Controls and domain restricted sharing

You configured CMEK as a default resource setting for your organization or for a folder, or you created a log bucket with CMEK enabled. You then configure VPC Service Controls. After you configure VPC Service Controls, you restrict access to Cloud Key Management Service in VPC Service Controls or enable domain restricted sharing.

At least one of the following occurs:

You received a notification from Cloud Logging about CMEK access issues.
You notice that CMEK isn't enabled for the _Default and _Required log buckets when you create new Google Cloud projects in your organization or in a folder.

You receive errors when you read from log buckets with CMEK enabled. The errors you see are similar to the following error:

ERROR: (gcloud.logging.read) FAILED_PRECONDITION: service account `cmek-PROJECT_IDgcp-sa-logging.iam.gserviceaccount.com` must have both encrypt and decrypt access to the CMEK KMS key `projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY`

You receive errors when you create or update log buckets with CMEK enabled. The errors you see are similar to the following error:

ERROR: (gcloud.logging.buckets.create) service account `cmek-PROJECT_ID@gcp-sa-logging.iam.gserviceaccount.com` must have both encrypt and decrypt access to the CMEK KMS key `projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY`
- '@type': type.googleapis.com/google.rpc.DebugInfo
  detail: '[ORIGINAL ERROR] generic::permission_denied: Request is prohibited by
  organization's policy. vpcServiceControlsUniqueIdentifier: <var>ERRORID</var>;'

To determine if these problems are due to the VPC Service Controls configuration, do the following:

Identify the Cloud Logging settings for the resource that contains the CMEK configuration. A resource can be a project, a folder, or an organization. If you created log buckets with CMEK enabled, then select the PROJECT resource.
PROJECT
```
gcloud logging settings describe --project=PROJECT_ID
```
Before you run the command, replace PROJECT_ID with the project ID that contains the log bucket.
FOLDER
```
gcloud logging settings describe --folder=FOLDER_ID
```
Before you run the command, replace FOLDER_ID with the ID of the folder.
ORGANIZATION
```
gcloud logging settings describe --organization=ORGANIZATION_ID
```
Before you run the command, replace ORGANIZATION_ID with the ID of the organization.
The previous command returns information similar to the following:
```
kmsServiceAccountId: KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
loggingServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
```
For organizations and folders, the following field is also returned:
```
kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY
```
The value of the kmsKeyName field include thes Google Cloud project that stores the key.
Determine whether you need to migrate service accounts:
- If the value of the kmsServiceAccountId field has the prefix service-, then you don't need to migrate your service account. For information about CMEK configuration errors, see the Troubleshoot CMEK section of this document.
- If the value of kmsServiceAccountId has the prefix cmek-, then proceed to the next step.
Confirm that you must migrate service accounts by disabling domain restricted sharing or by removing Cloud Key Management Service from the list of VPC Service Controls restricted services.

If the errors are resolved, then to resolve the failures you must migrate the affected resources to a new service account. For information about these steps, see the next section.

Migrate CMEK service accounts

The following process describes how to change the service account that Cloud Logging uses to access configured Cloud Key Management Service keys. The change in service account resolves a known issue with VPC Service Controls and domain restricted sharing.

Identify the loggingServiceAccountId for your resource. A resource can be a project, a folder, or an organization. If you created log buckets with CMEK enabled, then select the PROJECT resource.
PROJECT
```
gcloud logging settings describe --project=PROJECT_ID
```
Before you run the command, replace PROJECT_ID with the project ID that contains the log bucket.
FOLDER
```
gcloud logging settings describe --folder=FOLDER_ID
```
Before you run the command, replace FOLDER_ID with the ID of the folder.
ORGANIZATION
```
gcloud logging settings describe --organization=ORGANIZATION_ID
```
Before you run the command, replace ORGANIZATION_ID with the ID of the organization.
The previous command returns information similar to the following:
```
kmsServiceAccountId: KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
loggingServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
```
For organizations and folders, the following field is also returned:
```
kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY
```
The value of the kmsKeyName field include thes Google Cloud project that stores the key.
If you configured a default resource setting for your organization or for folders, then do the following:
1. In the KMS_PROJECT_ID, grant the Cloud Key Management Service CryptoKey Encrypter/Decrypter role to the service account identified by the loggingServiceAccountId field.
2. Run the following curl command, which changes the Cloud Key Management Service service account used by the resource.
  PROJECT
  Not applicable.
  FOLDER
```
curl -X PATCH -H "Authorization: Bearer \"$(gcloud auth print-access-token)\"" -H "Content-Type: application/json; charset=utf-8" -d '{"kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}' https://logging.googleapis.com/v2/folders/FOLDER_ID/settings?updateMask=kmsServiceAccountId
```
  Before you run the command, do the following:
  - Replace FOLDER_ID with the ID of the folder.
  - Replace SERVICE_ACCT_NAME with the loggingServiceAccountId identified earlier.
  ORGANIZATION
```
curl -X PATCH -H "Authorization: Bearer \"$(gcloud auth print-access-token)\"" -H "Content-Type: application/json; charset=utf-8" -d '{"kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}' https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings?updateMask=kmsServiceAccountId
```
  Before you run the command, do the following:
  - Replace ORGANIZATION_ID with the ID of the organization.
  - Replace SERVICE_ACCT_NAME with the loggingServiceAccountId identified earlier.
  The result of the previous command is similar to the following;
```
{
  "name": ".../settings",
  "kmsKeyName": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY",
  "kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com",
  "storageLocation": "...",
  "loggingServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"
}
```
Note: The new service account is used for new resources created in the organization or folder. The service account for existing resources isn't modified.
For each Google Cloud project or folder that contain existing log buckets enabled with CMEK, do the following:
1. In the project or folder, for each log bucket enabled with CMEK, do the following:
  1. Identify the Google Cloud project that stores the Cloud Key Management Service key:
    PROJECT
    gcloud logging buckets describe BUCKET_ID --location=LOCATION --project=PROJECT_ID
    
    Before you run the command, do the following:
    
    Replace PROJECT_ID with the project ID that contains the log bucket.
    
    Replace LOCATION with the location of the log bucket.
    FOLDER
    gcloud logging buckets describe BUCKET_ID --location=LOCATION --folder=FOLDER_ID
    
    Before you run the command, do the following:
    
    Replace FOLDER_ID with the ID of the folder.
    
    Replace LOCATION with the location of the log bucket.
    The result of the previous command is similar to the following:
```
cmekSettings:
  kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY
  kmsKeyVersionName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY/cryptoKeyVersions/1
  serviceAccountId: KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
createTime: '2022-10-31T12:00:00.0000000Z'
lifecycleState: ACTIVE
name: projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID
retentionDays: 30
createTime: '2022-10-31T13:00:00.0000000Z'
```
  2. Go to the Google Cloud project that owns the Cloud Key Management Service key, KMS_PROJECT_ID, and grant the Cloud Key Management Service CryptoKey Encrypter/Decrypter role to the service account identified by the loggingServiceAccountId field.
2. For the project, run the following curl command, which changes the Cloud Key Management Service service account:
  PROJECT
```
curl -X PATCH -H "Authorization: Bearer \"$(gcloud auth print-access-token)\"" -H "Content-Type: application/json; charset=utf-8" -d '{"kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}' https://logging.googleapis.com/v2/projects/PROJECT_ID/settings?updateMask=kmsServiceAccountId
```
  Before you run the command, do the following:
  - Replace PROJECT_ID with the project ID that contains the log bucket.
  - Replace SERVICE_ACCT_NAME with the loggingServiceAccountId identified earlier.
  FOLDER
  No action necessary, as you changed the Cloud Key Management Service service account used by the folder in a previous step.
  The result of the previous command is similar to the following;
```
{
  "name": ".../settings",
  "kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com",
  "loggingServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"
}
```
For the each log bucket with CMEK enabled, do the following:
1. Rotate the Cloud KMS key.
2. Confirm the migration. The parent resource for the log bucket determines which Google Cloud CLI command to run. The parent might be a project, folder, or organization.
  PROJECT
```
gcloud logging buckets describe BUCKET_ID --location=LOCATION --project=PROJECT_ID
```
  Before you run the command, do the following:
  - Replace PROJECT_ID with the project ID that contains the log bucket.
  - Replace LOCATION with the location of the log bucket.
  FOLDER
```
gcloud logging buckets describe BUCKET_ID --location=LOCATION --folder=FOLDER_ID
```
  Before you run the command, do the following:
  - Replace FOLDER_ID with the ID of the folder.
  - Replace LOCATION with the location of the log bucket.
  For a project, the result of the previous command is similar to the following:
```
cmekSettings:
  kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY
  kmsKeyVersionName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY/cryptoKeyVersions/1
  serviceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
createTime: '2022-10-31T12:00:00.0000000Z'
lifecycleState: ACTIVE
name: projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID
retentionDays: 30
createTime: '2022-10-31T13:00:00.0000000Z'
```
  Ensure that serviceAccountId matches the loggingServiceAccountId identified earlier.
Wait at least 30 minutes before revoking permissions on the previous service account. If you experience any issues after revoking permissions on the previous service account, then restore the permissions and contact Cloud Support.

Troubleshoot CMEK

As you configure CMEK, the Google Cloud project containing the Cloud KMS key is notified of related issues. For example, updates fail when the KMS_KEY_NAME is invalid, when the associated service account doesn't have the required Cloud Key Management Service CryptoKey Encrypter/Decrypter role, or when access to the key is disabled.

After you configure CMEK, at least on of the following occurs:

You received a notification from Cloud Logging about CMEK access issues.
You notice that CMEK isn't enabled for the _Default and _Required log buckets when you create new Google Cloud projects in your organization or in a folder.
You receive errors when you read from log buckets with CMEK enabled, or if you try to create or update log buckets.

The notification provides information about the failure and it contains that you can take to mitigate the issue:

Error	Recommendation
Cryptographic key permission denied	The Logging service account associated with your Google Cloud project doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. Follow the instructions in the error or see the following documents: If you have enabled VPC Service Controls, then see the Troubleshoot VPC Service Controls and domain restricted sharing. section of this document. Assign the Encrypter/Decrypter role for log buckets Assign the Encrypter/Decrypter role for organizations and folders
Cryptographic key is disabled	The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key.
Cryptographic key was destroyed	The specified Cloud KMS key was destroyed. Follow the instructions or see the following documents: Manage your Cloud KMS key for a log bucket Manage your Cloud KMS key for a folder or organization

Error

Recommendation

Cryptographic key permission denied

The Logging service account associated with your Google Cloud project doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. Follow the instructions in the error or see the following documents:

If you have enabled VPC Service Controls, then see the Troubleshoot VPC Service Controls and domain restricted sharing. section of this document.
Assign the Encrypter/Decrypter role for log buckets
Assign the Encrypter/Decrypter role for organizations and folders

Cryptographic key is disabled

The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key.

Cryptographic key was destroyed

The specified Cloud KMS key was destroyed. Follow the instructions or see the following documents:

Identify project that contains the Cloud KMS key

To identify the ID of the Google Cloud project that contains the encryption key used by a log bucket, folder, or organization, do the following:

PROJECT

gcloud logging settings describe --project=PROJECT_ID

Before you run the command, replace PROJECT_ID with the project ID that contains the log bucket.

FOLDER

gcloud logging settings describe --folder=FOLDER_ID

Before you run the command, replace FOLDER_ID with the ID of the folder.

ORGANIZATION

gcloud logging settings describe --organization=ORGANIZATION_ID

Before you run the command, replace ORGANIZATION_ID with the ID of the organization.

The previous command returns information similar to the following:

kmsServiceAccountId: KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
loggingServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com

For organizations and folders, the following field is also returned:

kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY

The value of the kmsKeyName field include thes Google Cloud project that stores the key.

Verify key usability

To verify the key's usability, run the following command to list all keys:

gcloud kms keys list \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING

This command returns information about each key in a tabular format. The first line of the output is a list of column names:

NAME PURPOSE ...

Verify that the Cloud KMS key is listed in the command's output as ENABLED, and that the purpose of the key is symmetric encryption: the PURPOSE column must contain ENCRYPT_DECRYPT and the PRIMARY_STATE column must contain ENABLED.

If necessary, create a new key.

Verify permissions configuration

Service accounts that are associated with the organization's CMEK settings must have the Cloud KMS CryptoKey Encrypter/Decrypter role for the configured key.

To list the key's IAM policy, run the following command:

gcloud kms keys get-iam-policy KMS_KEY_NAME

If necessary, add the service account that contains the Cloud KMS CryptoKey Encrypter/Decrypter role to the key.