Config Connector: An easy way to manage your infrastructure in Google Cloud

Today many companies manage their infrastructure and configure environments using multiple tools that are either stand-alone or a part of a larger CI/CD pipeline solution. Tools such as Cloud Build in Google Cloud, HashiCorp’s Terraform, or AWS CloudFormation, allow developers to use purpose-built languages such as HashiCorp’s HCL or Cloud Build Configuration language to define their environment's infrastructure and/or automate its provisioning.

One of the popular tools, Terraform, is a widely used Infrastructure as Code (IaC) software tool to provision infrastructure on Google Cloud and other cloud platforms. Google is actively supporting Terraform by contributing to the Google Cloud Provider for Terraform and developing the Cloud Foundation Toolkit which includes many useful Terraform modules.

We would like to evaluate another solution called Config Connector (a.k.a., KCC), available on Google Cloud, and to show how cloud users can improve their operational processes using this solution compared with other available tools. Google announced it first in 2020. Config Connector is a Kubernetes operator that allows you to manage Google Cloud resources. Config Connector utilizes the Kubernetes Resource Model to enforce a contract between the configuration a developer has defined and infrastructure. This is often referred to as Configuration as Data. You can read more about Configuration as Data in this blog post. Compared to Terraform, Config Connector applies a reconciliation strategy to keep cloud infrastructure as close to the declared configuration as possible in real time.

Config Connector can provide a developer with a number of advantages:

• Native integration with GKE and Anthos Configuration Management simplifies provisioning of both Google Cloud resources and application workloads across multiple environments.

• Automated reconciliation observes the infrastructure state and repairs any discrepancies between the desired and observed states without need for additional monitoring or manual intervention.

• Centralized configuration management lets you manage workload and infrastructure configurations for all environments in one place and in one format.

• As a managed solution, Config Connector reduces operational and maintenance overload on DevOps teams, saving time and helping to speed up onboarding new team members.

You can reference the following decision tree when deciding which tool to use when provisioning Google Cloud infrastructure:

Using Config Connector also lets developers benefit from extensive observability capabilities. Leveraging integration with GKE and Cloud Operations suite, you can audit Config Connector operations and the reconciliation state of the configuration. Additionally, you can automate incident handling by defining alert policies to be triggered when there are problems with configuration, provisioning or reconciliation. For example, the following set of log filters can be used to query problems with configuration references (e.g., a resource references a Kubernetes Secret that cannot be found):

See the Config Connector documentation about monitoring and troubleshooting for more information.

Getting started with Config Connector is simple. All you need is a GKE cluster. Then, you can enable the Config Connector add-on to have Config Connector automatically installed on the cluster. There are several options to install Config Connector. The following paragraphs summarizes pros and cons of each option.

• Config Controller is a great choice if you are looking to minimize maintenance cost and add support for GitOps components. To use it, you would have to enable Anthos in your projects which may introduce management and cluster fees. If you already use Anthos Config Management (ACM), Config Controller is already available for you. ACM hosts Config Connector and automatically upgrades it to the latest stable version.

• Manual installation is useful when you need a high level of customization and control over Config Connector. Using this method you install a Kubernetes operator and additional CRDs on your GKE cluster. This also enables you to install Config Connector on other Kubernetes distributions. It comes at higher operational costs since you will own the hosting and configuration of Config Connector.

• GKE Config Connector add-on is a good choice as a jump start solution. It can be installed on any new or existing GKE Standard cluster (starting version 1.15) using a single configuration setting. However, we would like to discourage you using it in production because of the significant lag behind the latest Config Connector version. It also comes with operational costs of provisioning and maintaining the hosting GKE cluster.

Once Config Connector is installed, you can provision Google Cloud resources like you do your Kubernetes workloads. For example, the following code snippet will create a BigQuery dataset:

(This example uses the user-specified resource ID to identify the BigQuery dataset)

In many scenarios Config Connector can replace multiple other tools while minimizing the time it takes to reconcile configured and actual states. The managed nature of the Connector together with a large coverage of Google Cloud resources and services, as well as integration with Anthos configuration, makes it a universal Swiss Army Knife of DevOps pipelines for Google Cloud users. You can familiarize yourself with Config Connector by reading the documentation. Give it a try!