title | description | author | tags | date_published |
---|---|---|---|---|
Customizing Fluent Bit for Google Kubernetes Engine logs |
Learn how to customize Fluent Bit for Google Kubernetes Engine logs. |
xiangshen-dk |
logging, stackdriver, gke, fluent-bit |
2020-11-25 |
Xiang Shen | Solutions Architect | Google
Contributed by Google employees.
This tutorial describes how to customize Fluent Bit logging for a Google Kubernetes Engine cluster. In this tutorial, you learn how to host your own configurable Fluent Bit daemonset to send logs to Cloud Logging, instead of selecting the Cloud Logging option when creating the Google Kubernetes Engine (GKE) cluster, which does not allow configuration of the Fluent Bit daemon.
This tutorial assumes that you're familiar with Kubernetes.
This tutorial applies to Linux nodes only.
Unless otherwise noted, you enter all commands for this tutorial in Cloud Shell.
- Deploy your own Fluent Bit daemonset on a Google Kubernetes Engine cluster, configured to log data to Cloud Logging.
- Customize GKE logging to remove sensitive data from the Cloud Logging logs.
This tutorial uses billable components of Google Cloud, including a three-node Google Kubernetes Engine cluster.
The pricing calculator estimates the cost of this environment at around $1.14 for 8 hours.
-
In the Cloud Console, on the project selector page, select or create a Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish this tutorial, you can delete the project, removing all resources associated with the project.
-
Make sure that billing is enabled for your Google Cloud project.
Learn how to confirm whether billing is enabled for your project.
-
Enable the Google Kubernetes Engine and Compute Engine APIs.
In this section, you define variables that control where elements of the infrastructure are deployed.
-
Set the variables used by this tutorial:
export region=us-east1 export zone=${region}-b export project_id=[YOUR_PROJECT_ID]
This tutorial uses the region
us-east-1
. If you change the region, make sure that the zone values reference your region. -
Set the default zone and project ID so that you don't have to specify these values in every subsequent command:
gcloud config set compute/zone ${zone} gcloud config set project ${project_id}
-
Clone the sample repository:
git clone https://github.com/GoogleCloudPlatform/community.git
The sample repository includes the Kubernetes manifests for the Fluent Bit daemonset and a test logging program that you deploy.
-
Go to the directory for this tutorial in the cloned repository:
cd community/tutorials/kubernetes-engine-customize-fluentbit
-
Create the GKE cluster with system-only logging turned on:
gcloud container clusters create custom-fluentbit \ --zone $zone \ --logging=SYSTEM \ --tags=gke-cluster-with-customized-fluentbit \ --scopes=logging-write,storage-rw
By default, the sample application that you deploy continuously emits random logging statements. The Docker container is
built from the source code under the test-logger
subdirectory.
-
Build the
test-logger
container image:docker build -t test-logger test-logger
-
Tag the container before pushing to the registry:
docker tag test-logger gcr.io/${project_id}/test-logger
-
Push the container image:
docker push gcr.io/${project_id}/test-logger
-
Update the deployment file:
envsubst < kubernetes/test-logger.yaml > kubernetes/test-logger-deploy.yaml
-
Deploy the
test-logger
application to the GKE cluster:kubectl apply -f kubernetes/test-logger-deploy.yaml
-
View the status of the
test-logger
pods:kubectl get pods
-
Repeat this command until the output looks like the following, with all three
test-logger
pods running:NAME READY STATUS RESTARTS AGE test-logger-58f7bfdb89-4d2b5 1/1 Running 0 28s test-logger-58f7bfdb89-qrlbl 1/1 Running 0 28s test-logger-58f7bfdb89-xfrkx 1/1 Running 0 28s
In this section, you configure and deploy your Fluent Bit daemonset.
Because you turned on system-only logging, a GKE-managed Fluentd daemonset is deployed that is responsible for system logging. The Kubernetes manifests for Fluent Bit that you deploy in this procedure are versions of the ones available from the Fluent Bit site for logging using Cloud Logging and watching changes to Docker log files.
-
Create the service account and the cluster role in a new
logging
namespace:kubectl apply -f ./kubernetes/fluentbit-rbac.yaml
-
Deploy the Fluent Bit configuration:
kubectl apply -f kubernetes/fluentbit-configmap.yaml
-
Deploy the Fluent Bit daemonset:
kubectl apply -f kubernetes/fluentbit-daemonset.yaml
-
Check that the Fluent Bit pods have started:
kubectl get pods --namespace=logging
-
If they're running, you see output like the following:
NAME READY STATUS RESTARTS AGE fluent-bit-246wz 1/1 Running 0 26s fluent-bit-6h6ww 1/1 Running 0 26s fluent-bit-zpp8q 1/1 Running 0 26s
For details of configuring Fluent Bit for Kubernetes, see the Fluent Bit manual.
-
Verify that you're seeing logs in Cloud Logging. In the console, on the left-hand side, select Logging > Logs Explorer, and then select Kubernetes Container as a resource type in the Resource list.
-
Click Run Query.
-
In the Logs field explorer, select test-logger for CONTAINER_NAME. After you add the
log
field to the summary line, you should see logs similar to the following:
In this section, you configure Fluent Bit to filter certain data so that it is not logged. For this tutorial, you filter out Social Security numbers, credit card numbers, and email addresses. To make this update, you change the daemonset to use a different ConfigMap that contains these filters. You use Kubernetes rolling updates feature and preserve the old version of the ConfigMap.
- Open the
kubernetes/fluentbit-configmap.yaml
file in an editor. - Uncomment the lines after
### sample log scrubbing filters
and before### end sample log scrubbing filters
. - Change the name of the ConfigMap from
fluent-bit-config
tofluent-bit-config-filtered
by editing themetadata.name
field. - Save and close the file.
In this section, you change kubernetes/fluentbit-daemonset.yaml
to mount the fluent-bit-config-filtered
ConfigMap instead of the
fluent-bit-config
ConfigMap.
-
Open the
kubernetes/fluentbit-daemonset.yaml
file in an editor. -
Change the name of the ConfigMap from
fluent-bit-config
tofluent-bit-config-filtered
by editing theconfigMap.name
field:- name: fluent-bit-etc configMap: name: fluent-bit-config
-
Deploy the new version of the ConfigMap to your cluster:
kubectl apply -f kubernetes/fluentbit-configmap.yaml
-
Roll out the new version of the daemonset:
kubectl apply -f kubernetes/fluentbit-daemonset.yaml
-
Roll out the update and wait for it to complete:
kubectl rollout status ds/fluent-bit --namespace=logging
When it completes, you should see the following message:
daemon set "fluent-bit" successfully rolled out
-
When the rollout is complete, refresh the Cloud Logging logs and make sure that the Social Security number, credit card number, and email address data has been filtered out.
If you don't want to delete the whole project, run the following command to delete the GKE cluster:
gcloud container clusters delete custom-fluentbit --zone us-east1-b
- Review Fluent Bit documentation in more detail.
- Review Google Kubernetes Engine documentation in more detail.
- Try out other Google Cloud features for yourself. Have a look at our tutorials.