Multi-tenant logging on GKE

This page explains how to configure multi-tenant logging for Google Kubernetes Engine (GKE) clusters.

It's common for multiple teams to share a single GKE cluster. Sharing a cluster provides multiple advantages including easier service discovery, simplified security, and it means that cluster administrators have fewer clusters to maintain. Individual application teams often have their own separate project, however. This structure, having a main GKE cluster, but separate namespaces for each application team, is called multi-tenancy. The application team's project is called the tenant.

With Google Cloud, GKE cluster admins can create a system where logs for the cluster remain in the main GKE project, and tenant logs are distributed to tenant projects. To configure your logs this way, utilize the Cloud Logging Logs Router. Logs Router gives you control over how logs flow within your Google Cloud project and how logs flow to other Google Cloud projects.

To create tenant-specific logs, the cluster admin creates a sink to export log entries to a log bucket created in the tenant's project. You can also, optionally, create an exclusion rule to prevent tenant logs being stored in the main GKE project.

The following diagram provides an overview of a multi-tenant logging architecture using log buckets:

GKE multi-tenant architecture

This architecture includes:

  1. A log bucket created in each tenant project.
  2. A log sink created for each tenant namespace.
  3. A service account automatically created for each log sink.
  4. Exclusion rules to, optionally, prevent duplication of logs in the main GKE project.

The following sections guide you through how to create such an architecture.

Prerequisites

Configuring multi-tenant logging

You can configure multi-tenant logging using the gcloud command-line tool or the Google Cloud Console.

gcloud

To implement multi-tenant logging for GKE clusters, complete the following steps:

  1. Set variables:

    export TENANT_NAMESPACE="TENANT_NAMESPACE"
    export MAIN_PROJECT="MAIN_PROJECT_ID"
    export TENANT_PROJECT="TENANT_PROJECT_ID"
    

    Replace the following:

    • TENANT_NAMESPACE: the name for the tenant project namespace
    • MAIN_PROJECT_ID: the project ID for your main project
    • TENANT_PROJECT_ID: the project ID for your tenant project
  2. Create a namespace in your multi-tenant cluster:

    kubectl create namespace $TENANT_NAMESPACE
    
  3. Create the log bucket in the tenant project:

    gcloud logging buckets create gke-$TENANT_NAMESPACE-log-bucket \
        --project=$TENANT_PROJECT \
        --location=global \
        --description="Log bucket for $TENANT_NAMESPACE namespace from $MAIN_PROJECT"
    

    For more information about these fields, see the gcloud logging buckets create API documentation.

  4. Create a log sink in the main GKE project:

    gcloud logging sinks create gke-$TENANT_NAMESPACE-sink \
    logging.googleapis.com/projects/$TENANT_PROJECT/locations/global/buckets/gke-$TENANT_NAMESPACE-log-bucket \
        --project=$MAIN_PROJECT \
        --log-filter=resource.labels.namespace_name="$TENANT_NAMESPACE" \
        --description="Log sink to $TENANT_PROJECT for $TENANT_NAMESPACE namespace"
    

    This command creates a log sink that sends all logs that are related to the $TENANT_NAMESPACE namespace to the log bucket you created in the previous step.

    Occasionally, you might need to use a more restrictive --log-filter. For example, if your cluster and tenant have the same namespace, you could add a cluster filter.

    For more information about these fields, see the gcloud logging sinks create API documentation.

  5. Get the service account from the sink in the main project and assign it to a variable. You need this service account to grant permissions in the next step.

    export SERVICE_ACCOUNT=$(gcloud logging sinks describe gke-$TENANT_NAMESPACE-sink \
        --project=$MAIN_PROJECT \
        --format='value(writerIdentity)')
    
  6. Grant the logging.bucketWriter role to the service account used by the sink. The main project needs this permission to write to the bucket in the tenant project.

     gcloud projects add-iam-policy-binding $TENANT_PROJECT \
         --member=$SERVICE_ACCOUNT --role='roles/logging.bucketWriter' \
         --condition="expression=resource.name.endsWith(\"locations/global/buckets/gke-$TENANT_NAMESPACE-log-bucket\"),title=Log bucket writer for $TENANT_NAMESPACE,description=Grants logging.bucketWriter role to service account $SERVICE_ACCOUNT used by gke-$TENANT_NAMESPACE-sink"
    

    For more information about these fields, see the gcloud projects add-iam-policy-binding API documentation.

  7. Optionally, create an exclusion rule in the _Default bucket. This rule stops the tenant logs from also being written to the main bucket. If you omit this command, you will have duplicate logs in the main project's _Default bucket and the tenant bucket.

    gcloud logging sinks update _Default --project=$MAIN_PROJECT \
        --add-exclusion="name=gke-$TENANT_NAMESPACE-default-exclusion,description=\"Exclusion rule on the _Default bucket for $TENANT_NAMESPACE\",filter=resource.labels.namespace_name=\"$TENANT_NAMESPACE\""
    

    For more information on these fields, see the gcloud logging sinks update API documentation.

Console

To implement multi-tenant logging for GKE, complete the following steps:

  1. Create a log bucket in the tenant project:

    1. Go to the Logs Storage menu:

      Go to Logs Storage

    2. Click the project dropdown at the top page and select the tenant project.

    3. Click Create logs bucket.

    4. Enter a Name and Description for your bucket.

    5. From the Select Logs Bucket Region dropdown, select a region.

    6. Click Create bucket. Your new bucket appears in the Logs bucket list.

  2. Create the log sink in the main GKE project:

    1. Click the project dropdown at the top page and select the main GKE project.
    2. In the left-hand menu, select Logs Router. You are taken to the Logs Router page.
    3. Click Create sink. The Select sink window appears.
    4. In the Select sink window, select Cloud Logging bucket.
    5. Enter a Name and Description for your sink and click Next.
    6. In the Select sink service drop-down, select Other project.
    7. In the Sink destination field, add the following destination: logging.googleapis.com/projects/MAIN_PROJECT_ID/locations/LOG_BUCKET_REGION/buckets/BUCKET_NAME

      Replace the following:

      • MAIN_PROJECT_ID: the project ID for your main project
      • LOG_BUCKET_REGION: the region that you created your log bucket in
      • BUCKET_NAME: the name of the log bucket you created in the previous section
    8. Click Next.

    9. In the Build inclusion filter add the following filter: resource.labels.namespace_name="TENANT_NAMESPACE". Replace TENANT_NAMESPACE with the name of your tenant project's namespace.

      Occasionally, you might need to use a more restrictive inclusion filter. For example, if your cluster and tenant have the same namespace, you could add a cluster filter.

    10. Click Create sink. Your new sink appears in the Logs Routing Sinks list.

  3. Get the service account from the sink in the main project. You need this service account to grant permissions in the next step.

    1. In the Logs Router page, locate the log sink from the main project.
    2. Next to that sink, click More and select View sink details.
    3. Copy the value next to Writer Identity: serviceAccount:.
  4. Grant the Log Bucket Writer role to the service account used by the tenant's sink. The main project needs this permission to write to the bucket in the tenant project.

    1. In the Cloud Console, go to the IAM page.

      Go to IAM

    2. Click Add.

    3. In the New members field, add the sink's service account.

    4. From the Select a role drop-down, select Logging and choose Log Bucket Writer.

    5. Click Save.

  5. Optionally, create an exclusion rule in the _Default bucket. This stops the tenant logs from also being written to the main bucket. If you omit this command, you will have duplicate logs in the main project's _Default bucket and the tenant bucket.

    1. In the Cloud Console, go to the Logs Router page.

      Go to Logs Router

    2. Next to the _Default bucket, click More and select Edit sink.

    3. In the Choose logs to filter out of sink section, click Add exclusion.

    4. Add a filter name.

    5. In the Build an exclusion filter box, add resource.labels.namespace_name=\"TENANT_NAMESPACE"\

    6. Click Update sink.

Verifying tenant logs

After you have begun using workloads that use TENANT_NAMESPACE, you can verify that the tenant project is receiving tenant-specific logs:

  1. From the tenant project, go to the Logs Viewer page in the Cloud Console.

    Go to Logs Viewer

  2. Click Refine scope.

  3. Select Scope by storage and select the tenant's bucket:

    gke-TENANT_NAMESPACE-log-bucket
    

Cleaning up

You can remove the objects you created for multi-tenant logging using gcloud or the Cloud Console.

gcloud

To remove the objects you created for multi-tenant logging, complete the following steps:

  1. Set variables to simplify the following commands:

    export TENANT_NAMESPACE="TENANT_NAMESPACE"
    export MAIN_PROJECT="MAIN_PROJECT_ID"
    export TENANT_PROJECT="TENANT_PROJECT_ID"
    

    Replace the following:

    • TENANT_NAMESPACE: the name for the tenant project namespace
    • MAIN-PROJECT-ID: the project ID for your main project
    • TENANT-PROJECT-ID: the project ID for your tenant project
  2. If you created an exclusion rule in the main project, remove it:

    gcloud logging sinks update _Default \
       --project=$MAIN_PROJECT \
       --remove-exclusions=gke-$TENANT_NAMESPACE-default-exclusion
    
  3. Remove the bucketWriter role from the service account:

    export SERVICE_ACCOUNT=$(gcloud logging sinks describe gke-$TENANT_NAMESPACE-sink \
        --project=$MAIN_PROJECT | \
        --format='value(writerIdentity)'
    
    gcloud projects remove-iam-policy-binding $TENANT_PROJECT \
        --member=$SERVICE_ACCOUNT \
        --role='roles/logging.bucketWriter' \
        --all
    
  4. Delete the log sink:

    gcloud logging sinks delete gke-$TENANT_NAMESPACE-sink \
        --project=$MAIN_PROJECT
    
  5. Delete the log bucket:

    gcloud logging buckets delete gke-$TENANT_NAMESPACE-log-bucket \
        --project=$TENANT_PROJECT \
        --location=global
    
  6. Delete the namespace:

    kubectl delete namespace $TENANT_NAMESPACE
    

Console

  1. If you created an exclusion rule in the main project, remove it:

    1. In the Cloud Console, go to the Logs Router page.

      Go to Logs Router

    2. Next to the _Default bucket, click More

    3. Select Edit sink.

    4. Next to the exclusion rule you created, click Delete.

    5. Click Update sink.

  2. In the main project, remove the service account:

    1. In the Cloud Console, go to the IAM page.

      Go to IAM

    2. Select the sink's service account.

    3. Click Remove.

    4. In the confirmation window, click Confirm.

  3. In the tenant project, delete the log sink:

    1. Click the project dropdown at the top page and select the tenant GKE project.
    2. From the Logging menu, select Logs Router.

      Go to Logs Router

    3. For the sink you want to delete, click More .

    4. Select Delete sink.

    5. In the confirmation panel, click Delete.

  4. In the main project, delete the log bucket:

    1. Click the project dropdown at the top page and select the main GKE project.
    2. From the Logging menu, select Logs Storage.

      Go to Logs Storage

    3. For the bucket you want to delete, click More .

    4. Select Delete bucket.

    5. On the confirmation panel, click Delete.

Limitations

Multi-tenant logging has the following limitations:

  • The quota for the number of log sinks per project is 200. If you require more than 200 tenants, request a quota uplift by opening a support case.
  • There is a hard limit of 50 exclusion rules per log bucket. If you intend to have more than 50 tenants, the exclusion rule approach for the _Default bucket needs to be revised. As an alternative, you could:

    • Create a single exclusion rule that filters out all non-system or non-default namespaces using this command:

      gcloud logging sinks update _Default \
      --project=$MAIN_PROJECT \
      --add-exclusion="name=gke-all-tenant-default-exclusion,description=\"Exclusion rule on the _Default bucket for all tenants\",filter=resource.labels.namespace_name !~ \"kube\" AND resource.labels.namespace_name !~ \"system\ AND resource.labels.namespace_name != \"Default\""
      
    • Duplicate logs between the tenant project and the main project by not creating the exclusion rule.

What's next