Multi-tenant logging on GKE

This page explains how to configure multi-tenant logging for Google Kubernetes Engine (GKE) clusters.

Overview

It's common for multiple teams to share a single GKE cluster. Sharing a cluster provides multiple advantages including easier service discovery, simplified security, and it means that cluster administrators have fewer clusters to maintain. Individual application teams often have their own separate project, however. This structure, having a main GKE cluster, but separate namespaces for each application team, is called multi-tenancy. The application team's project is called the tenant.

With Google Cloud, GKE cluster admins can create a system where logs for the cluster remain in the main GKE project, and tenant logs are distributed to tenant projects. To configure your logs this way, utilize the Cloud Logging Logs Router. Logs Router gives you control over how logs flow within your Google Cloud project and how logs flow to other Google Cloud projects.

To create tenant-specific logs, the cluster admin creates a sink to export log entries to a log bucket (beta) created in the tenant's project. You can also, optionally, create an exclusion rule to prevent tenant logs being stored in the main GKE project.

The following diagram provides an overview of a multi-tenant logging architecture using log buckets:

GKE multi-tenant architecture

This architecture includes:

  1. A log bucket created in each tenant project.
  2. A log sink created for each tenant namespace.
  3. A service account automatically created for each log sink.
  4. Exclusion rules to, optionally, prevent duplication of logs in the main GKE project.

The following sections guide you through how to create such an architecture.

Prerequisites

Configuring multi-tenant logging

You can configure multi-tenant logging using gcloud or the Google Cloud Console.

gcloud

To implement multi-tenant logging for GKE clusters, complete the following steps:

  1. Optionally, set variables and create a namespace for the tenant GKE cluster:

    export TENANT_NAME="tenant-name"
    export MAIN_PROJECT="main-project-id"
    export TENANT_PROJECT="tenant-project-id"
    kubectl create namespace $TENANT_NAME
    
  2. Create the log bucket (beta) in the tenant project:

    gcloud alpha logging buckets create tenant-bucket-name \
        --project=tenant-project-id \
        --location=location \
        --description=description
    

    Replace the following:

    • tenant-bucket-name: The name of your tenant log bucket. Give the bucket a descriptive name to make configuration easier. For example, you could name it gke-$TENANT_NAME-log-bucket.
    • tenant-project-id: Your tenant project ID.
    • location: The location to create the bucket in. Once the bucket is created, the location cannot be changed.
    • description: (Optional) a description to clarify this bucket's purpose. For example, "Log bucket for $TENANT_NAME namespace from $MAIN_PROJECT".

    For more information about these fields, see the gcloud alpha logging buckets create API documentation.

  3. Create a log sink in the main GKE project:

    gcloud alpha logging sinks create main-sink-name sink-destination \
        --project=main-project-id \
        --log-filter=resource.labels.namespace_name="tenant-name" \
        --description=description
    

    Replace the following:

    • main-sink-name: The name of your sink. Give the sink a descriptive name to make configuration easier. For example, you could name it gke-$TENANT_NAME-sink.
    • sink-destination: The location of the bucket you created in the previous step. The bucket location has the following format: logging.googleapis.com/projects/tenant-project-id/locations/global/buckets/bucket-name.
    • main-project-id: Your main project ID.
    • resource.labels.namespace_name=tenant-namespace: This filter causes the sink to export all of the tenant logs.
    • description: (Optional) a description to clarify this bucket's purpose. For example, "Log sink to $TENANT_PROJECT for $TENANT_NAME namespace"

    For more information about these fields, see the gcloud alpha logging sinks create API documentation.

  4. Get the service account from the sink in the main project. You need this service account to grant permissions in the next step.

    # Set a variable to use in the next step
    export SERVICE_ACCOUNT=gcloud alpha logging sinks describe main-sink-name \
        --project=main-project-id | \
    grep serviceAccount | \
    sed 's/.*\(serviceAccount:.*\)/\1/'
    
  5. Grant the logging.bucketWriter role to the service account used by the sink. The main project needs this permission to write to the bucket in the tenant project.

    gcloud projects add-iam-policy-binding tenant-project-id \
        --member=$SERVICE_ACCOUNT \
        --role='roles/logging.bucketWriter' \
    # Add a condition to limit the access only to writing logs.
        --condition="expression=resource.name.endsWith\
        (\"locations/global/buckets/tenant-bucket-name\"),\
        title=Log bucket writer for tenant"
    

    For more information about these fields, see the gcloud projects add-iam-policy-binding API documentation.

  6. Optionally, create an exclusion rule in the \_Default bucket. This stops the tenant logs from also being written to the main bucket. If you omit this command, you will have duplicate logs in the main project's \_Default bucket and the tenant bucket.

    gcloud alpha logging sinks update _Default \
      --project=main-project-id \
      --add-exclusion="name=gke-tenant-default-exclusion,\
       description=\"Exclusion rule on the _Default bucket for tenant\",\
       filter=resource.labels.namespace_name=\"tenant-namespace\""
    

    For more information on these fields, see the gcloud alpha logging sinks update API documentation.

Console

To implement multi-tenant logging for GKE, complete the following steps:

  1. Create a log bucket (beta) in the tenant project:

    1. Go to the Logs Storage menu:

      Go to Logs Storage

    2. Click the project dropdown at the top page and select the tenant project.

    3. Click Create logs bucket.

    4. Enter a Name and Description for your bucket.

    5. Click Create bucket. Your new bucket appears in the Logs bucket list.

  2. Create the log sink in the main GKE project:

    1. Click the project dropdown at the top page and select the main GKE project.
    2. In the left-hand menu, select Logs Router. You are taken to the Logs Router page.
    3. Click Create sink. The Select sink window appears.
    4. In the Select sink window, select Cloud Logging.
    5. Enter a Name and Description for your sink and click Next.
    6. In the Select sink service drop-down, select Cloud Logging.
    7. In the Select logs bucket drop-down, select the tenant logs bucket you created in the previous step and click Next.
    8. In the Build inclusion filter add the following filter: resource.labels.namespace_name="tenant-namespace". Replace tenant-project-id with the name of your tenant project.
    9. Click Create sink. Your new sink appears in the Logs Routing Sinks list.
  3. Get the service account from the sink in the main project. You need this service account to grant permissions in the next step.

    1. In the Logs Router page, locate the log sink from the main project.
    2. Next to that sink, click More and select View sink details.
    3. Copy the value next to Writer Identity: serviceAccount:.
  4. Grant the Log Bucket Writer role to the service account used by the tenant's sink. The main project needs this permission to write to the bucket in the tenant project.

    1. In the Cloud Console, go to the IAM page.

      Go to IAM

    2. Click Add.

    3. In the New members field, add the sink's service account.

    4. From the Select a role drop-down, select Logging and choose Log Bucket Writer.

    5. Click Save.

  5. Optionally, create an exclusion rule in the \_Default bucket. This stops the tenant logs from also being written to the main bucket. If you omit this command, you will have duplicate logs in the main project's \_Default bucket and the tenant bucket.

    1. In the Cloud Console, go to the Logs Router page.

      Go to Logs Router

    2. Next to the \_Default bucket, click and select Edit sink.

    3. In the Choose logs to filter out of sink section, click Add exclusion.

    4. Add a filter name.

    5. In the Build an exclusion filter box, add resource.labels.namespace_name=\"tenant-namespace"\

    6. Click Update sink.

Verifying tenant logs

To verify that the tenant project is receiving tenant-specific logs:

  1. From the tenant project, go to the Logs Viewer page.

    Go to Logs Viewer

  2. Click Refine scope.

  3. Select Scope by storage and select the tenant's bucket.

  4. Run queries for logs. For example, resource.labels.namespace_name="tenant-namespace".

Cleaning up

You can remove the objects you created for multi-tenant logging using gcloud or the Cloud Console.

gcloud

To remove the objects you created for multi-tenant logging, complete the following steps:

  1. Set variables to simplify the following commands:

    export TENANT_NAME="tenant-project-id"
    export MAIN_PROJECT="main-project-id"
    export TENANT_PROJECT="tenant-project-id"
    
  2. If you created an exclusion rule in the main project, remove it:

    gcloud alpha logging sinks update _Default \
    --project=$MAIN_PROJECT \
    --remove-exclusions=gke-$TENANT_NAME-default-exclusion
    
  3. Remove the bucketWriter role from the service account:

    export SERVICE_ACCOUNT=`gcloud alpha logging sinks describe \
    gke-$TENANT_NAME-sink --project=$MAIN_PROJECT | \
    grep serviceAccount | \
    sed 's/.*\(serviceAccount:.*\)/\1/'`
    
    gcloud projects remove-iam-policy-binding $TENANT_PROJECT \
    --member=$SERVICE_ACCOUNT --role='roles/logging.bucketWriter'
    
  4. Delete the log sink:

    gcloud alpha logging sinks delete gke-$TENANT_NAME-sink \
      --project=$MAIN_PROJECT
    
  5. Delete the log bucket (beta):

    gcloud alpha logging buckets delete gke-$TENANT_NAME-log-bucket \
      --project=$TENANT_PROJECT \
      --location=global
    
  6. Delete the namespace:

    kubectl delete namespace $TENANT_NAME
    

Console

  1. If you created an exclusion rule in the main project, remove it:

    1. In the Cloud Console, go to the Logs Router page.

      Go to Logs Router

    2. Next to the \_Default bucket, click More

    3. Select Edit sink.

    4. Next to the exclusion rule you created, click Delete.

    5. Click Update sink.

  2. In the main project, remove the service account:

    1. In the Cloud Console, go to the IAM page.

      Go to IAM

    2. Select the sink's service account.

    3. Click Remove.

    4. In the confirmation window, click Confirm.

  3. In the tenant project, delete the log sink:

    1. Click the project dropdown at the top page and select the tenant GKE project.
    2. From the Logging menu, select Logs Router.

      Go to Logs Router

    3. For the sink you want to delete, click More .

    4. Select Delete sink.

    5. In the confirmation panel, click Delete.

  4. In the main project, delete the log bucket (beta):

    1. Click the project dropdown at the top page and select the main GKE project.
    2. From the Logging menu, select Logs Storage.

      Go to Logs Storage

    3. For the bucket you want to delete, click More .

    4. Select Delete bucket.

    5. On the confirmation panel, click Delete.

Limitations

Multi-tenant logging has the following limitations:

  • The quota for the number of log sinks per project is 200. If you require more than 200 tenants, request a quota uplift by opening a support case.
  • There is a hard limit of 50 exclusion rules per log bucket. If you intend to have more than 50 tenants, the exclusion rule approach for the _Default bucket needs to be revised. As an alternative, you could:

    • Create a single exclusion rule that filters out all non-system or non-default namespaces using this command:

      gcloud alpha logging sinks update _Default \
      --project=$MAIN_PROJECT \
      --add-exclusion="name=gke-all-tenant-default-exclusion,description=\"Exclusion rule on the _Default bucket for all tenants\",filter=resource.labels.namespace_name !~ \"kube\" AND resource.labels.namespace_name !~ \"system\ AND resource.labels.namespace_name != \"Default\""
      
    • Duplicate logs between the tenant project and the master project by not creating the exclusion rule.

What's next