Customizing Fluent Bit for Google Kubernetes Engine logs
Contributed by Google employees.
This tutorial describes how to customize Fluent Bit logging for a Google Kubernetes Engine cluster. In this tutorial, you learn how to host your own configurable Fluent Bit daemonset to send logs to Cloud Logging, instead of selecting the Cloud Logging option when creating the Google Kubernetes Engine (GKE) cluster, which does not allow configuration of the Fluent Bit daemon.
This tutorial assumes that you're familiar with Kubernetes.
This tutorial applies to Linux nodes only.
Unless otherwise noted, you enter all commands for this tutorial in Cloud Shell.
Objectives
- Deploy your own Fluent Bit daemonset on a Google Kubernetes Engine cluster, configured to log data to Cloud Logging.
- Customize GKE logging to remove sensitive data from the Cloud Logging logs.
Costs
This tutorial uses billable components of Google Cloud, including a three-node Google Kubernetes Engine cluster.
The pricing calculator estimates the cost of this environment at around $1.14 for 8 hours.
Before you begin
In the Cloud Console, on the project selector page, select or create a Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish this tutorial, you can delete the project, removing all resources associated with the project.
Make sure that billing is enabled for your Google Cloud project.
Learn how to confirm whether billing is enabled for your project.
Enable the Google Kubernetes Engine and Compute Engine APIs.
Initializing common variables
In this section, you define variables that control where elements of the infrastructure are deployed.
Set the variables used by this tutorial:
export region=us-east1 export zone=${region}-b export project_id=[YOUR_PROJECT_ID]
This tutorial uses the region
us-east-1
. If you change the region, make sure that the zone values reference your region.Set the default zone and project ID so that you don't have to specify these values in every subsequent command:
gcloud config set compute/zone ${zone} gcloud config set project ${project_id}
Creating the GKE cluster
Clone the sample repository:
git clone https://github.com/GoogleCloudPlatform/community.git
The sample repository includes the Kubernetes manifests for the Fluent Bit daemonset and a test logging program that you deploy.
Go to the directory for this tutorial in the cloned repository:
cd community/tutorials/kubernetes-engine-customize-fluentbit
Create the GKE cluster with system-only logging turned on:
gcloud container clusters create custom-fluentbit \ --zone $zone \ --logging=SYSTEM \ --tags=gke-cluster-with-customized-fluentbit \ --scopes=logging-write,storage-rw
Deploying the test logger application
By default, the sample application that you deploy continuously emits random logging statements. The Docker container is
built from the source code under the test-logger
subdirectory.
Build the
test-logger
container image:docker build -t test-logger test-logger
Tag the container before pushing to the registry:
docker tag test-logger gcr.io/${project_id}/test-logger
Push the container image:
docker push gcr.io/${project_id}/test-logger
Update the deployment file:
envsubst < kubernetes/test-logger.yaml > kubernetes/test-logger-deploy.yaml
Deploy the
test-logger
application to the GKE cluster:kubectl apply -f kubernetes/test-logger-deploy.yaml
View the status of the
test-logger
pods:kubectl get pods
Repeat this command until the output looks like the following, with all three
test-logger
pods running:NAME READY STATUS RESTARTS AGE test-logger-58f7bfdb89-4d2b5 1/1 Running 0 28s test-logger-58f7bfdb89-qrlbl 1/1 Running 0 28s test-logger-58f7bfdb89-xfrkx 1/1 Running 0 28s
Deploying the Fluent Bit daemonset to your cluster
In this section, you configure and deploy your Fluent Bit daemonset.
Because you turned on system-only logging, a GKE-managed Fluentd daemonset is deployed that is responsible for system logging. The Kubernetes manifests for Fluent Bit that you deploy in this procedure are versions of the ones available from the Fluent Bit site for logging using Cloud Logging and watching changes to Docker log files.
Create the service account and the cluster role in a new
logging
namespace:kubectl apply -f ./kubernetes/fluentbit-rbac.yaml
Deploy the Fluent Bit configuration:
kubectl apply -f kubernetes/fluentbit-configmap.yaml
Deploy the Fluent Bit daemonset:
kubectl apply -f kubernetes/fluentbit-daemonset.yaml
Check that the Fluent Bit pods have started:
kubectl get pods --namespace=logging
If they're running, you see output like the following:
NAME READY STATUS RESTARTS AGE fluent-bit-246wz 1/1 Running 0 26s fluent-bit-6h6ww 1/1 Running 0 26s fluent-bit-zpp8q 1/1 Running 0 26s
For details of configuring Fluent Bit for Kubernetes, see the Fluent Bit manual.
Verify that you're seeing logs in Cloud Logging. In the console, on the left-hand side, select Logging > Logs Explorer, and then select Kubernetes Container as a resource type in the Resource list.
Click Run Query.
In the Logs field explorer, select test-logger for CONTAINER_NAME. After you add the
log
field to the summary line, you should see logs similar to the following:
Filtering information from the log file
In this section, you configure Fluent Bit to filter certain data so that it is not logged. For this tutorial, you filter out Social Security numbers, credit card numbers, and email addresses. To make this update, you change the daemonset to use a different ConfigMap that contains these filters. You use Kubernetes rolling updates feature and preserve the old version of the ConfigMap.
- Open the
kubernetes/fluentbit-configmap.yaml
file in an editor. - Uncomment the lines after
### sample log scrubbing filters
and before### end sample log scrubbing filters
. - Change the name of the ConfigMap from
fluent-bit-config
tofluent-bit-config-filtered
by editing themetadata.name
field. - Save and close the file.
Updating the Fluent Bit daemonset to use the new configuration
In this section, you change kubernetes/fluentbit-daemonset.yaml
to mount the fluent-bit-config-filtered
ConfigMap instead of the
fluent-bit-config
ConfigMap.
- Open the
kubernetes/fluentbit-daemonset.yaml
file in an editor. Change the name of the ConfigMap from
fluent-bit-config
tofluent-bit-config-filtered
by editing theconfigMap.name
field:- name: fluent-bit-etc configMap: name: fluent-bit-config
Deploy the new version of the ConfigMap to your cluster:
kubectl apply -f kubernetes/fluentbit-configmap.yaml
Roll out the new version of the daemonset:
kubectl apply -f kubernetes/fluentbit-daemonset.yaml
Roll out the update and wait for it to complete:
kubectl rollout status ds/fluent-bit --namespace=logging
When it completes, you should see the follwoing message:
daemon set "fluent-bit" successfully rolled out
When the rollout is complete, refresh the Cloud Logging logs and make sure that the Social Security number, credit card number, and email address data has been filtered out.
Cleaning up: deleting the GKE cluster
If you don't want to delete the whole project, run the following command to delete the GKE cluster:
gcloud container clusters delete custom-fluentbit --zone us-east1-b
What's next
- Review Fluent Bit documentation in more detail.
- Review Google Kubernetes Engine documentation in more detail.
- Try out other Google Cloud features for yourself. Have a look at our tutorials.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.