Hybrid and multi-cloud container image management with Harbor
Jani Patokallio | Solutions Architect | Google
Contributed by Google employees.
This tutorial describes how to implement a multi-site container registry design with Harbor and Google Container Registry. In this tutorial, you set up one-way, push-based replication of container images from Harbor to Google Container Registry.
It's becoming increasingly common to need to deploy a single application system across multiple sites. These different sites could be different public clouds (multi-cloud) or a combination of public and on-premises data centers (hybrid cloud).
Having container registries local to the Kubernetes clusters at multiple sites speeds the retrieval of application container images by decreasing latency and increasing bandwidth, and it prevents a single container registry from being a single point of failure for the application running across independent Kubernetes clusters.
Using different container registries for different clusters has a downstream impact, because the Kubernetes deployment manifests deployed to each cluster need to use different fully qualified image names. This challenge can be addressed with different templating tools (such Helm, kpt, or Kustomize) but this is out of scope for this tutorial. A simple solution using shell scripting is provided at the end of this tutorial for reference and adaptation.
Note: As of May 2021, Google Artifact Registry is not supported by Harbor as an external registry endpoint.
Install Harbor and its prerequisites
This tutorial uses a single-node installation of Harbor on Compute Engine. You can use any Compute Engine virtual machine with at least 2 vCPUs, with Allow HTTPS traffic enabled under Firewall.
Note: For a highly available installation on Anthos instead, you can use the instructions for deploying Harbor with high availability with Helm, which also requires setting up HA clusters of PostgreSQL and Redis.
Install the prerequisites for Harbor, including Docker Engine and Docker Compose.
Follow the instructions for installing and configuring Harbor.
For the Harbor HTTPS configuration, use the internal Compute Engine hostname as your fully qualified domain name (FQDN). When you're logged in to the VM, set the environment variable
$HARBORto this value with the following command:HARBOR=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/hostname" -H "Metadata-Flavor: Google")After installation is complete, test that you can access the Harbor user interface by browsing to
https://[VM_EXTERNAL_IP_ADDRESS].The default username is
admin. The default password isadmin/Harbor12345.Confirm that you can log in to Docker with the same credentials:
docker login $HARBORThis command should return
Login Successful.
Set up the Google Container Registry endpoint for Harbor
- In the Cloud Console, go to the IAM & Admin > Service Accounts page and
create a Google Cloud service account with the
Storage Adminrole. - Click the key name and select the Keys tab.
Under Add key, select Create a new key and JSON.
This creates a new key and downloads it to your local machine.
In Harbor, go to Administration > Registries > New Endpoint and fill in the following details in the Edit Endpoint dialog box:
- Provider: Choose Google GCR.
- Name: Your endpoint name. This tutorial uses
gcr-sea-anthos-demo. - Description: A description for your endpoint.
- Endpoint URL:
https://gcr.io - Access ID:
_json_key - Access Secret: Paste your Google Cloud IAM service account JSON key content in this field.
- Verify Remote Cert: Check this box.
Click Test Connection.
Click OK.
Set up replication
In Harbor, go to Administration > Replications > New Replication Rule, fill in the following details in the Edit Replication Rule dialog box, and then click Save:
- Name: Your replication rule name. This tutorial uses
replicate-library-to-gcr. - Description: A description for your replication rule.
- Replication Mode: Select Push-based, which means that Harbor is the primary container registry that new images are pushed to, so replication to other sites should be triggered when new images are pushed to Harbor.
Source resource filter:
- Name:
[LIBRARY_NAME]/**. For example, if you uselibrary/**as the filter, all images underlibraryare replicated.
For more example filter patterns, see Creating a replication rule. - Tag: (optional) Filter based on image tag/version. - Label: (optional) Filter based on Harbor labels. - Resource: Image. Only replicate container images to Google Container Registry.
- Name:
Destination registry: Name of the registry endpoint that you created in the previous section.
Destination namespace:
[YOUR_GOOGLE_CLOUD_PROJECT_NAME]/[OPTIONAL_GOOGLE_CONTAINER_REGISTRY_LIBRARY_NAME]. The Google Cloud project name is mandatory. If you also specify a subdirectory with the library name, then all source images will be stored in this subdirectory.Trigger Mode: Select Event Based.
Delete remote resources when locally deleted: Check this box to keep Google Container Registry fully synchronized.
Override: Check this box to replace (overwrite) an image at the destination when an image of the same name exists.
Enable rule: Check this box.
Harbor does not support subdirectories under the library. If a container is stored in Harbor with .../... in the name, the subdirectory name will be
removed. For example, library/private/nginx will be replicated to [DESTINATION_NAMESPACE]/nginx, with the private subdirectory dropped.
Test replication
To test replication, pull a public nginx image, tag it, and push it to Harbor:
$docker pull nginx:latest
$docker tag nginx:latest $HARBOR/library/private-reg/nginx:latest
$docker push $HARBOR/library/private-reg/nginx:latest
The image is automatically replicated to Google Container Registry.
The screenshots below show the state of Harbor and Google Container Registry after a new image is pushed to the local Harbor registry, and automatic synchronization is successful.
Successful execution in Harbor:
State of Google Container Registry:
State of Harbor registry:
Updating image registry path in image name
You can use the example code in this section to replace repository URLs in fully qualified image names.
This example replaces registry.private.com/lib/nginx:v1.1 with gcr.io/lib/nginx:v1.1:
# export OLD_IMG="registry.private.com/lib/nginx:v1.1"
$ export NEW_REPO="gcr.io\/lib\/"
$ NEW_IMG=`echo $OLD_IMG | sed "s/^\(.*\/\)\(.*\)/${NEW_REPO}\2/"`
$ echo $NEW_IMG
gcr.io/lib/nginx:v1.1
What's next
For more information, see the Harbor documentation about configuring replication.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.