You are viewing documentation for Anthos Service Mesh 1.10. View the latest documentation or select another available version:

Migrating to Mesh CA

Migrating to Anthos Service Mesh certificate authority (Mesh CA) from Istio CA (also known as Citadel) requires migrating the root of trust. Prior to Anthos Service Mesh 1.10, if you wanted to migrate from Istio on Google Kubernetes Engine (GKE) to Anthos Service Mesh with Mesh CA, you needed to schedule downtime because Anthos Service Mesh was not able to load multiple root certificates. Therefore, during the migration, the newly deployed workloads trust the new root certificate, while others trust the old root certificate. Workloads using certificates signed by different root certificates can't authenticate with each other. This means that mutual TLS (mTLS) traffic is interrupted during the migration. The entire cluster only fully recovers when the control plane and all workloads in all namespaces are redeployed with Mesh CA's certificate. If your mesh has multiple clusters with workloads that send requests to workloads on another cluster, all workloads on those clusters need to be updated as well.

This page describes how to migrate from Istio CA to Mesh CA with minimal or no downtime. Use the steps in this guide for the following use cases:

  • Migrate from Istio 1.9 or 1.10 on GKE to the Anthos Service Mesh 1.10.4-asm.14 in-cluster control plane with Mesh CA.
  • Upgrade from Anthos Service Mesh 1.9 or a 1.10 patch release with Istio CA to the Anthos Service Mesh 1.10.4-asm.14 in-cluster control plane with Mesh CA.

Limitations

  • Mesh CA is supported only on GKE.
  • All GKE clusters must be in the same Google Cloud project.

Prerequisites

This guide assumes that you have the following:

Required tools

During the migration, you run a Google-provided script, migrate_ca, to validate the following for each Pod in the cluster:

  • The root certificate for Istio CA and Mesh CA.
  • The workload mTLS certificate issued by Istio CA and by Mesh CA.
  • The trust domains configured by Istio CA and Mesh CA.

This script has the following dependencies:

  • awk
  • grep
  • istioctl When you run the install_asm script, it downloads the version of istioctl that matches the version of Anthos Service Mesh that you are installing.
  • jq
  • kubectl
  • openssl

Overview of the migration

To migrate to Mesh CA, you follow the revision-based migration process (also referred to as a "canary upgrade"). With a revision-based migration, a new control plane revision is deployed alongside the existing control plane. You then gradually move your workloads to the new revision, which lets you monitor the effect of the migration through the process. During the migration process, authentication and authorization are fully functional between workloads using the Mesh CA and workloads using the Istio CA.

Following is an outline of the migration to Mesh CA:

  1. Distribute the Mesh CA root of trust.

    1. Install a new control plane revision with an option that will distribute the Mesh CA root of trust.

    2. Migrate workloads to the new control plane, namespace by namespace, and test your application. When all workloads are successfully migrated to the new control plane, remove the old control plane.

  2. Migrate to Mesh CA. Now that all sidecar proxies are configured with the old root of trust and the Mesh CA root of trust, you can migrate to Mesh CA without downtime. Again, you follow the revision-based migration process:

    1. Install a control plane revision with Mesh CA enabled.

    2. Migrate workloads to the new control plane revision, namespace by namespace, and test your application. When all workloads are successfully migrated to the new control plane, remove the old control plane.

    3. Remove CA secrets in the cluster that are associated with the old CA and restart the new control plane.

Distribute the Mesh CA root of trust

Before you can migrate to Mesh CA, all GKE clusters in the mesh must have Anthos Service Mesh 1.10 or later, and all clusters must be configured with a control plane option that triggers the root of trust for Mesh CA to be distributed to the proxies of all workloads on the cluster. When the process is finished, each proxy is configured with both the old and the new root of trusts. With this scheme, when you migrate to Mesh CA, workloads using Mesh CA will be able to authenticate with workloads using the old CA.

Install a new control plane revision

Install a control plane revision with an option that distributes the Mesh CA root of trust.

  1. Follow the steps in Installing Anthos Service Mesh on GKE to get ready to use a Google-provided script, install_asm, to install the new control plane revision.

  2. Make sure you have the version of install_asm that installs Anthos Service Mesh 1.10 or higher:

    ./install_asm --version
    
  3. Run install_asm. In the following command, replace the placeholders with your values.

    • PROJECT_ID: Required. The project ID of the project that the cluster was created in.

    • CLUSTER_NAME: Required. The name of the cluster.

    • CLUSTER_LOCATION: Required. Either the zone or region that the cluster is in.

    • REVISION_1: Recommended. A revision label is a key-value pair that is set on the control plane. The revision label key is always istio.io/rev. By default, the script sets the value for the revision label based on the Anthos Service Mesh version, for example: asm-1104-14. We recommend that you include this option and replace REVISION_1 with a name that describes the installation, such as asm-1104-14-distribute-root. The name must be a DNS-1035 label, and it must consist of lower case alphanumeric characters or -, start with an alphabetic character, and end with an alphanumeric character (such as my-name', or abc-123). The regex used for validation is: '[a-z]([-a-z0-9]*[a-z0-9])?')

    • DIR_PATH : Required. A relative path to a directory where the script downloads the anthos-service-mesh package and the Anthos Service Mesh installation file, which contains istioctl, samples, and manifests.

    • OVERLAYS: Optional. If you want to enable optional features, include --custom_overlay with the name of the overlay file for each feature. If you aren't enabling optional features, delete this line and the backslash on the preceding line.

      ./install_asm \
        --project_id  PROJECT_ID \
        --cluster_name CLUSTER_NAME \
        --cluster_location CLUSTER_LOCATION \
        --mode install \
        --ca citadel \
        --enable_all \
        --option ca-migration-citadel \
        --revision_name REVISION_1  \
        --output_dir DIR_PATH \
        OVERLAYS
      

    The following command-line arguments are required:

    • --ca citadel: To avoid downtime, specify Istio CA (the citadel option corresponds to Istio CA). Don't switch to Mesh CA at this point.

    • --option ca-migration-citadel: When you redeploy your workloads, this option triggers the new root of trust to be distributed to the sidecar proxies of the workloads.

Migrate workloads to the new control plane

To finish distributing the new root of trust, you need to label your namespaces with the revision label istio.io/rev=asm-1104-14-distribute-root and restart your workloads. When testing your workloads after restarting them, you run a run a script to validate that the sidecar proxy is configured with both the old and new root of trust for Mesh CA.

  1. Set the current context for kubectl. In the following command, change --region to --zone if you have a single-zone cluster.

    gcloud container clusters get-credentials CLUSTER_NAME \
        --project=PROJECT_ID \
        --region=CLUSTER_LOCATION
    
  2. Download the validation script:

    curl https://raw.githubusercontent.com/GoogleCloudPlatform/anthos-service-mesh-packages/release-1.10-asm/scripts/ca-migration/migrate_ca > migrate_ca
    
  3. Set the executable bit on the script:

    chmod +x migrate_ca
    
  4. The migrate_ca script calls istioctl, which is version dependent. The install_asm script adds a symlink to istioctl in the directory you specified for --output_dir. Make sure that directory is at the beginning of your path. In the following command, replace ISTIOCTL_PATH with the directory that contains istioctl that the script downloaded.

    export PATH=ISTIOCTL_PATH:$PATH
    which istioctl
    echo $PATH
    
  5. Get the revision label that is on istiod and the istio-ingressgateway.

    kubectl get pod -n istio-system -L istio.io/rev
    

    The output is similar to the following:

    NAME                                                             READY   STATUS    RESTARTS   AGE   REV
    istio-ingressgateway-5fd454f8ff-t7w9x                            1/1     Running   0          36m   default
    istio-ingressgateway-asm-195-2-distribute-root-c6ccfbdbd-z2s9p   1/1     Running   0          18m   asm-195-2-distribute-root
    istio-ingressgateway-asm-195-2-distribute-root-c6ccfbdbd-zr2cs   1/1     Running   0          18m   asm-195-2-distribute-root
    istiod-68bc495f77-shl2h                                          1/1     Running   0          36m   default
    istiod-asm-195-2-distribute-root-6f764dbb7c-g9f8c                1/1     Running   0          18m   asm-195-2-distribute-root
    istiod-asm-195-2-distribute-root-6f764dbb7c-z7z9s                1/1     Running   0          18m   asm-195-2-distribute-root
    1. In the output, under the REV column, note the value of the revision label for the new revision, which matches the revision label that you specified when you ran install_asm. In this example, the value is asm-1104-14-distribute-root.

    2. You need to delete the old revision of istiod when you finish moving workloads to the new revision. Note the value in the revision label for the old istiod revision. The example output shows a migration from Istio, which is using the default revision.

  6. Switch the istio-ingressgateway to the new revision. In the following command, make sure that REVISION_1 matches the value of the revision label of the new revision.

    kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "REVISION_1"}]'

    Expected output:

    service/istio-ingressgateway patched
  7. Add the revision label to a namespace and remove the istio-injection label (if it exists). In the following command, replace NAMESPACE with the namespace to label.

    kubectl label namespace NAMESPACE istio.io/rev=REVISION_1 istio-injection- --overwrite

    If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

  8. Restart the Pods to trigger re-injection.

    kubectl rollout restart deployment -n NAMESPACE
    
  9. Verify that your Pods are configured to point to the new version of istiod.

    kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION_1
    
  10. Test your application to verify that the workloads are working correctly.

  11. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.

  12. Validate that the sidecar proxies for all workloads on the cluster are configured with both the old and new root certificates:

    ./migrate_ca check-root-cert
    

    Expected output:

    Namespace: foo
    httpbin-66cdbdb6c5-pmzps.foo trusts [CITADEL MESHCA]
    sleep-64d7d56698-6tmjm.foo trusts [CITADEL MESHCA]
  13. If you are satisfied that your application is working as expected, continue with the steps to transition to the new version of istiod. If there's an issue with your application, follow the steps to rollback.

    Complete the transition

    If you are satisfied that your application is working as expected, remove the old control plane to complete the transition to the new version.

    1. Change to the directory where the files from the anthos-service-mesh GitHub repository are located.

    2. Configure the validating webhook to use the new control plane.

      kubectl apply -f asm/istio/istiod-service.yaml
      
    3. Delete the old istio-ingressgatewayDeployment. The command that you run depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh:

      Migrate

      If you migrated from Istio, the old istio-ingressgateway doesn't have a revision label.

      kubectl delete deploy/istio-ingressgateway -n istio-system
      

      Upgrade

      If you upgraded from a previous Anthos Service Mesh version, in the following command, replace OLD_REVISION with the revision label for the previous version of the istio-ingressgateway.

      kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=OLD_REVISION -n istio-system --ignore-not-found=true
      
    4. Delete the old revision of istiod. The command that you use depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh.

      Migrate

      If you migrated from Istio, the old istio-ingressgateway doesn't have a revision label.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true
      

      Upgrade

      If you upgraded from a previous Anthos Service Mesh version, in the following command, make sure that OLD_REVISION matches the revision label for the previous version of istiod.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-OLD_REVISION -n istio-system --ignore-not-found=true
      
    5. Remove the old version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-OLD_REVISION -n istio-system
      

      The expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-OLD_REVISION" deleted

    Rollback

    If you encountered an issue when testing your application with the new version of istiod, follow these steps to rollback to the previous version:

    1. Switch back to the old version of the istio-ingressgatewqy. In the following command, replace OLD_REVISION with the old revision.

      kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "OLD_REVISION"}]'
      
    2. Relabel your namespace to enable auto-injection with the previous version of istiod. The command that you use depends on whether you used a revision label or istio-injection=enabled with the previous version.

      • If you used a revision label for auto-injection:

        kubectl label namespace NAMESPACE istio.io/rev=OLD_REVISION --overwrite
        
      • If you used istio-injection=enabled:

        kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite
        

      Expected output:

      namespace/NAMESPACE labeled
    3. Confirm that the revision label on the namespace matches the revision label on the previous version of istiod:

      kubectl get ns NAMESPACE --show-labels
      
    4. Restart the Pods to trigger re-injection so the proxies have the previous version:

      kubectl rollout restart deployment -n NAMESPACE
      
    5. Remove the new istio-ingressgateway Deployment.

      kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=REVISION_1 -n istio-system --ignore-not-found=true
      
    6. Remove the new revision of istiod.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-REVISION_1 -n istio-system --ignore-not-found=true
      
    7. Remove the new IstioOperator configuration.

      kubectl delete IstioOperator installed-state-asm-1104-14-distribute-root -n istio-system
      

      Expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-asm-1104-14-distribute-root" deleted

Migrate to Mesh CA

Now that the sidecar proxies for all workloads are configured with both the old root of trust and the new root of trust for Mesh CA, the steps to migrate to Mesh CA are similar to those that you did to distribute the Mesh CA root of trust:

Install a new control plane with Mesh CA enabled

You use install_asm to install a new control plane revision that has Mesh CA enabled.

  1. If you customized the previous installation, you need to specify the same overlay files when you run install_asm.

  2. Run install_asm. In the following command, replace the placeholders with your values.

    • PROJECT_ID: Required. The project ID of the project that the cluster was created in.

    • CLUSTER_NAME: Required. The name of the cluster.

    • CLUSTER_LOCATION: Required. Either the zone or region that the cluster is in.

    • REVISION_2: Recommended. Replace REVISION_2 with a name that describes the installation, such as asm-1104-14-meshca-ca-migration. The name must be a DNS-1035 label, and it must consist of lower case alphanumeric characters or -, start with an alphabetic character, and end with an alphanumeric character (such as my-name', or abc-123). The regex used for validation is: '[a-z]([-a-z0-9]*[a-z0-9])?')

    • DIR_PATH : Required. A relative path to a directory where the script downloads the anthos-service-mesh package and the Anthos Service Mesh installation file, which contains istioctl, samples, and manifests.

    • OVERLAYS: Optional. If you want to enable optional features, include --custom_overlay with the name of the overlay file for each feature. If you aren't enabling optional features, delete this line and the backslash on the preceding line.

    ./install_asm \
      --project_id  PROJECT_ID \
      --cluster_name CLUSTER_NAME \
      --cluster_location CLUSTER_LOCATION \
      --mode install \
      --ca mesh_ca \
      --enable_all \
      --option ca-migration-meshca \
      --revision_name REVISION_2 \
      --output_dir DIR_PATH \
      OVERLAYS
    

    The following command-line arguments are required:

    • --ca mesh_ca: You can now switch to Mesh CA since the Mesh CA root of trust has been distributed.

    • --option ca-migration-migration: When you redeploy your workloads, this option configures the proxies to use the Mesh CA root of trust.

Migrate workloads to the new control plane

To finish the installation, you need to label your namespaces with the new revision label and restart your workloads.

  1. Get the revision label that is on istiod and the istio-ingressgateway.

    kubectl get pod -n istio-system -L istio.io/rev
    

    The output is similar to the following:

    NAME                                                                          READY   STATUS    RESTARTS   AGE   REV
    istio-ingressgateway-asm-1104-14-distribute-root-65d884685d-6hrdk      1/1     Running   0          67m   asm-1104-14-distribute-root
    istio-ingressgateway-asm-1104-14-distribute-root65d884685d-94wgz       1/1     Running   0          67m   asm-1104-14-distribute-root
    istio-ingressgateway-asm-1104-14-meshca-ca-migration-8b5fc8767-gk6hb   1/1     Running   0          5s    asm-1104-14-meshca-ca-migration
    istio-ingressgateway-asm-1104-14-meshca-ca-migration-8b5fc8767-hn4w2   1/1     Running   0          20s   asm-1104-14-meshca-ca-migration
    istiod-asm-1104-14-distribute-root-67998f4b55-lrzpz                    1/1     Running   0          68m   asm-1104-14-distribute-root
    istiod-asm-1104-14-distribute-root-67998f4b55-r76kr                    1/1     Running   0          68m   asm-1104-14-distribute-root
    istiod-asm-1104-14-meshca-ca-migration-5cd96f88f6-n7tj9                1/1     Running   0          27s   asm-1104-14-meshca-ca-migration
    istiod-asm-1104-14-meshca-ca-migration-5cd96f88f6-wm68b                1/1     Running   0          27s   asm-1104-14-meshca-ca-migration
    1. In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1104-14-meshca-ca-migration.

    2. Also note the value in the revision label for the old istiod version. You need this to delete the old version of istiod when you finish moving workloads to the new version. In the example, the value of the revision label for the previous revision is `asm-1104-14-distribute-root.

  2. Switch the istio-ingressgateway to the new revision. In the following command, change REVISION_2 to the value that matches the revision label of the new version.

    kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "REVISION_2"}]'

    Expected output:

    service/istio-ingressgateway patched
  3. Add the new revision label to a namespace In the following command, replace NAMESPACE with the namespace to label.

    kubectl label namespace NAMESPACE istio.io/rev=REVISION_2 --overwrite
    
  4. Restart the Pods to trigger re-injection.

    kubectl rollout restart deployment -n NAMESPACE
    
  5. Verify that your Pods are configured to point to the new version of istiod.

    kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION_2
    
  6. Test your application to verify that the workloads are working correctly. Make sure that mTLS communication works between workloads in the older namespace and workloads in the newer namespace.

  7. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.

  8. If you are satisfied that your application is working as expected, continue with the steps to transition to the new control plane. If there's an issue with your application, follow the steps to rollback.

    Complete the transition

    If you are satisfied that your application is working as expected, remove the old control plane to complete the transition to the new version.

    1. Change to the directory where the files from the anthos-service-mesh GitHub repository are located.

    2. Configure the validating webhook to use the new control plane.

      kubectl apply -f asm/istio/istiod-service.yaml
      
    3. Delete the old istio-ingressgatewayDeployment. In the following command, replace OLD_REVISION with the revision label for the previous version of the istio-ingressgateway.

      kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=OLD_REVISION -n istio-system --ignore-not-found=true
      
    4. Delete the old istiod revision. In the following command, replace OLD_REVISION with the revision label for the previous version of istiod.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-OLD_REVISION -n istio-system --ignore-not-found=true
      
    5. Remove the old IstioOperator configuration.

      kubectl delete IstioOperator installed-state-OLD_REVISION -n istio-system
      

      The expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-OLD_REVISION" deleted

    Rollback

    If you encountered an issue when testing your application with the new istiod revision, follow these steps to rollback to the previous revision:

    1. Switch back to the previous istio-ingressgatewqy. In the following command, replace OLD_REVISION with the old revision.

      kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "OLD_REVISION"}]'
      
    2. Relabel your namespace to enable auto-injection with the previous istiod revision.

      kubectl label namespace NAMESPACE istio.io/rev=OLD_REVISION --overwrite
      

      Expected output:

      namespace/NAMESPACE labeled
    3. Confirm that the revision label on the namespace matches the revision label on the previous version of istiod:

      kubectl get ns NAMESPACE --show-labels
      
    4. Restart the Pods to trigger re-injection so the proxies have the previous version:

      kubectl rollout restart deployment -n NAMESPACE
      
    5. Remove the new istio-ingressgateway Deployment.

      kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=REVISION_2 -n istio-system --ignore-not-found=true
      
    6. Remove the new version of istiod. Make sure that the revision label in the following command matches your revision.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-REVISION_2 -n istio-system --ignore-not-found=true
      
    7. Remove the new version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-REVISION_2 -n istio-system
      

      Expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-REVISION_2" deleted

Remove the CA secrets and restart the new control plane

  1. Preserve secrets just in case you need them:

    kubectl get secret/cacerts -n istio-system -o yaml > save_file_1
    kubectl get secret/istio-ca-secret -n istio-system -o yaml > save_file_2
    
  2. Remove the CA secrets in the cluster associated with the old CA:

     kubectl delete secret cacerts istio-ca-secret -n istio-system --ignore-not-found
    
  3. Restart the newly installed control plane. This makes sure the old root of trust is cleaned up from all workloads running in the mesh.

    kubectl rollout restart deployment -n istio-system